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* 818984: (1) 
PREFACE 


A bit of history will be helpful to anyone who may be wondering 
just how this brochure happens to be in existence. Beginning in 1932 
the author has given annually a course in statistics at the Department 
of Agriculture Graduate School. The course consists of a survey of 
classical probability and an introduction to modern theoretical sta- 
tistics, with some philosophy mixed in with it in an attempt to 
separate the mathematical developments from observations on the phys- 
ical behavior of nature. The last six weeks of the course are de- 
voted to the study of least squares, 


It is the lecture notes for this part of the course for 1937 
that now appear. When the good and harm of their public appearance 
have been separated and weighed, whatever excess there be in the 
positive direction is to be credited to my friend Miss Besse B, Day 
of the Forest Service. In December 1937 she asked permission to 
reproduce my manuscript for the use of her colleagues in Washington 
and in the ficld, some of whom it has been my privilege to have worked 
with, in and out of classes. I pleaded for a few weeks in which to 
put the notes into better order, The task secmed hopeless; they had 
grown up not with the notion that they were to form a text, complete 
in itself, but simply to serve in the last part of the course described 
earlier, But after some thought in the matter, it became clear that 
there is a widespread need for just this type of material; accordingly 
Miss Day's original plan has been carried out. The assumption is that 
the reader knows when he needs to use least squares and when he should 
not, that is, when the question can be answered at all; In the course 
of study in which these notes have grown up, such things are discussed 
before least squares is commenced; the material presented here may 
therefore be regarded purely es an exposition on the principle and 
method. Naturally I have been partial to my own experience, which 
has embraced consultation on a wide variety of problems, mostly in 
the Department of Agriculture and the National Buresu of Standards, 

I have tried to stress points that have seemed inadequately covered 

or incorrectly pictured by previous writers, It is presumed that 
teachers using this as a text will heed the references for supplementary 
reading, and will also illustrate the theoretical exercises with 
numerical examples. 


During the years 1922-23 it was my good fortune to have had 

as a teacher Professor Charles A, Hutchinson of the University of 
Colorado; later, Professor Horace S. Uhler of Yale, to whose paper 

of 1923 I owe much of the inspiration for whatever is new in the 
present work, More recently I have had the privilege of either 
studying or working with R. A. Fisher, J. Neyman, and Egon S, Pearson 
in London, and Walter A, Shewhart of New York, thus to have been 
exposed to several different views of the ever lively question, how 
did we lcarn what we think we know, and what do we really know? 


Sa, 


(ii) 


Most of the computations in the following pages were performed 
by my wife, Lola s, Deming. The typing is the work of Miss Eleanor Wood 
of the Forest Service, whose skill and patience in writing mathematics 
will be appreciatea by the readers as much as it has been by the author, 
Equally valuable has been the assistance in checking and proof-reading 
rendered by Miss Day and Miss Marion M. Sandomire, also of the Forest 
Service; a number of blunders that they uncovered in the manuscript 
would have been to me a perennial source of embarrassment. A final and 
expert perusal of the stencils by Lee Garby (Mrs. €. D. G.), of the 
Bureau of Chemistry and Soils, has been no less contributory. Lastly, 
it is a pleasure to record an appreciation of my: counselor and chief, 
Or. Olas Kunsman, on account of his never failing encouragement and 
enthusiasm for mathematics during my eleven years in the government 
Service, ’ 


As the labor of putting these notes into print is being concluded, 
I am conscious of many failings, both by omission and commission; yet 
_the words of Thomas Simpson in the preface of the first edition of his 
Treatise of Algebra (London, 1745) give an accurate expression of my own 
thoughts: 


As to the Motives that first gave Birth to the ensuing 
Work, they were not so much any extravagant. Hopes I could 
form to myself of greatly extending the Subject by the 
Addition of a large Variety of new Improvements, though, 
the Reader will find many Things here moré than"he has yet 
seen, and perhaps than he expects, as an-earnest Desire of 
seeing a Subject of so much Importance established on a 
Clear and rational Foundation, ... - se a 


W. E. D, 


Washington ~ 
May 1938 
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Preliminary; some fragments in the history of least squares - - 1 


I. SOME SIMPLE PROBLEMS IN CURVE FITTING 
Peer OOO ett er Spm ee Cee rehome re ae ee me im a mem oD 
Pierav igs exenpletor curve ittingi.<\ streets ele Se 7 


The same problem with unequal weights 


(aj) The) solution’ for’ the’ parameter"a .~ ~ -)6 -s1 = = - = 9 
(oy The solution inj tabular form - --- - = 4 - - - = = - 10 
Avdigression to define weights ----+-- -~s 777 ett dik 


Several samples, all on the same unknown 


(a) All observations have the same precision - ----- - 15 
(b) The precisions of the single observations 

dit fersfrom onetesanmpie te anhthem = '- -— i 4si- ~~ LY 
(ce) {The estimates of G0 = - - - -\-4- = - ee me ee Ee 18 


(d) Comparison) of the two estimates; + -—- = - <---> - 4 


Another simple problem--the slope of a line that 
is known to pass through the origin 
(a) The y coordinates subject to error; x free 


OL error |: + =i eS) ae ye) eae > be 
(6b) The tabular calculation of b and its weight’e@: - -!- -'- 24 
(c) The x coordinates subject to error; y free 
DEL GTTOI Bee pe ee Se om eae aE eo 28) 
pee oa Dee Laon). OL er eee it ee tah mee tee ae) ats 
II. MCRE COMPLICATED PROBLEMS 
Bhesreneen. problem inv legen squares Ai om ese ol 


Short expressions for s*. The normal equations are really 
-normal, their coefficients symmetrical and positive } 
NE ES BS et ae bos? a a Mt a at lca ae 40 


Geometrical conditions, or no parameters 
faqihe mormal eqteatione = - = - = -9- = m4 t)- - o> - 4l 
(bo) The plane triangle continued. The weights of the 
Bazies anaeeetunction (or them =- =< - = =<-- = = = 47 


(iv) - 


12, Gsometrical conditions, continued 
(a) Systematic computation; the weights of functions 
of the adjusted values - ------+-+-+-+-+-+-- 
(b) Numerical example--a surveying problem - - - ---- - 
(c) Conelusions from the solution of the normal equations 
(d) Shorter method of computing the weights of a 
large number of functions - - - = = = = - = = == 


IIIT. APPLICATION TO CURVE FITTING 


13. Adjustable parameters: enter the conditions; the problem 


Of, COevas Lit ta ngs mi mt i | ee! Pac, ae ee 


14, Adjusting the observations, or finding the calculated points - 
15. Some geometry concerning the adjustment of observations - - - 


16, Systematic procedure for setting up the normal equations 


for the parametersy “+ + = 4 -- =). = See See eee 


17. Systematic solution of the normal equations, The reciprocal 
matrix. Systematics computation of Bef lawn ee le ay ee 


18. The weights of the parameters; their standard errors. The 
standard error of a function of the parameters, The 
standard error of a curve. The confidence bands associated 
With & CUDVO\ nm pie) ee el i Se i ee a 


19, Exercises and notes on the formation of normal equations 
fomvar loupe unct ine 4 lH =) Se ee 
The linen = jaca) axe Gee nee (Ree ee ee ee 
THe para bole em roe Ho) wet am ce cat el Ee eS ee 
The exponential and its logarithmic form/- -)92)~)- -.— oes 
The exponential with a linear componert ~~ “eo - = «i. c 
The generalized hyperbola and its logarithmic form - - - - 
The hyperbola with a linear component ----+-+-+-+-+-s. 
Mistelleneoisil Uae ee sole. ere See etn on, nee 


20. Three examples in curve fitting 
1. Fitting an isotherm = =~ - - - eee oe en 
«, The parabola y = a + bx + cx®, both coordinates 
In -ProTr ae SPE Gels) i) te det, ee 


a 
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Some Notes On 
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" |... quamobrem systema maxime probabile valorum pro 
guantitaribus p, @, 2B, 1s ecc, 1d erit, ubi agsregatum 
bhvv + h' h'vtv! + h'h'ytyt + ete, i.e. ubi summa 
quadratorum differentiarum inter valores revera 
observatos et computatos per numeros qui praecisionis 
gradum mctiuntur multiplicatarum fit minimum,” Karl 
Friedrich Gauss, Theoria Motus Corporum Coeicstiun, 
(Hamburg, 1809). Art. 179. 


1, Preliminary, some fragment s in the history of least squares, 
It is helpful in the study of any subject to keep an eye on its history. 
But as Max Nordau* has pointed out, history is elusive; there are er- 
rors of observation in history of a grosser kind than in the laboratory; 
moreover, there is always a personal judgment as to just what events 
in the past really contributed to later ones, In what follows, the 
numerous references to Gauss (1777-1855) will leave no doubt in the 
reader's mind that in my own judgment he accomplished most of what we 
think we know now about least squares. 


But of course there were others, A hundred years before Gauss’ 
time, De Moivre** originated the normal curve (1723), which was tabled 
by Kramp* in 1799, Multitudes since the time of Gauss have helped to 
clarify the basic principles involved by looking askance at theoretical 
statistics and in particular anything developed from the normal curve”. 


* Max Nordau, The Interpretation of History (Transl., Rebmen, London, 
1910). 


** Karl Pearson, Historical note, Biometrika 16, 402-404, 1924, A 
facsimile of the famous seven page pamphlet by De Moivre appears in 
an article by R. C. Archibald in Isis 8, October 1926. 


+ Kramp, Analyse des réfractions (Strasbourg, 1799), Some historical 
notes regarding this book will appear in an article by Mason DuPré in 
Isis, 1938, possibly 1959. 


++ Gauss himself cast a critical eye at it, but satisfied himself thst 
deductions from it must be useful, witness the following quotation 
from his Theorie Motus Art. 178: "Functio modo eruta omni quidem 


[See, 1] 


There. were Laplace and Legendre whom Gauss mentioned; Encke, Bessel, 
Airy, and Helmert, and I suppose many others, In 1879 came a neglected 
paper by Kummell (vide infra) treating the problem of curve fitting 
with great clarity. In 1877 Mansfield Merriman* made up a list of all 
the writings on least squares thet had come to his notice--408 books 
and papers altogether, He hardly hoped that the list could be conplete, 
but it is of great value. Of course he missed De Moivre, whose work 
was lost until Kerl Pearson uncovered it in 1924, 


Gauss, Bessel, Airy, and Helmert were astronomers, and in most 
of the problems cf adjustment that they met, the conditions imposed 
on the adjusted observations were geometrical, not involving adjustable 
parameters, and the result was that many questions regarding curve 
fitting were left for a later time. Kummell** in 1879 seems to have 
filled in a good share of what was missing, Unfortunately, his work 
fell on poor soil; the journel that he wrote in was obscure and his 
paper has had very littls influence on textbooks or general usage, He 
set down the condition equations for the problem of curve fitting when 
more than one coordinate is subject to error, and he remarked that the 
Situation was hopeless except in some of the simplest cases, one of 
which he proceeded to treat--the straight line, for which he obtained 
a solution under the supposition that the weights of the x and y co- 
ordinates are in a constant ratio for all points (see Ex. 6 of section 
19). He showed how the weights of the estimated parameters Alea c 
could be obtained from the normal equations along with the sum of the 
Squares of the weighted residuals, if ever the problem is not too com- 
plex to permit the equations to be set up. The use of the Lagrange 
multipliers, had he tried them, would have provided the general solu- 
tion that he was looking for (section 9 et seq.). 


Karl Pearson” in 1901 wrote "On lines and planes of closest 
fit to systems of points in space", Therein he obtained the best 
rigore errorum probabilitates exprimere certo non potest: quum enim 
errores possibiles semper limitibus certis coérceantur, errorum maiorum 
probabilitas semper evadere deberet = O, dum formula nostra semper 
valorem finitum exhibet. Attamen hic defectus, quo omnis functio 
analytica natura sue laborare debet, ad omnes usus practicos nullius 
momenti est, quum valor functionis nostrac tam rapide decrescat, 
quamprimum hA valorem considerabilen acquisivit, ut tuto ipsi 0 
aequivalens censeri possit. Praeteres ipsos errorum limites absoluto 
rigore assignare, rei natura numquam permittet". (His hA is propor- 
tional to the more usual x//20). 


* Mansfield Merriman, Connecticut Academy of Arts and Sciences 4, 
151-232, 1877-82, 


** Charles H, Kummell, The Analyst (Des Moines) 6, 97-105)" 1879. 


* Karl Pearson, Phil. Mag. (London) 2, 559-572, 1901. 


[Sec. 1] 


and worst fitting lines and planes, together with ingenious and now well 
known formulas for the mean square residual, the implicit premise being 

that all observations have the same weight (cf. Ex. 9 in seetion:19). 

Pearson did not make the assumption of small residuals; his results are 

exact. He evidently had not heard of Kummell's paper. 


Papers by Stewart* (1920) and Uhler**(1923) should be consulted 
for a careful explanation of the principle of least squares, especially 
in regard to curve fitting. In 1921 Corrado Ginit found the best and 
worst fitting lines for points whose x and y coordinates are in a 
constant ratio from point to point, the main results having already 
been reached by Kummell. In the same issue of Metron in which Gini 
wrote, Lowell J. Reed** described a clever mechanical device for fit- 
ting a line when the x and y coordinates have equal weights everywhere, 
all that is needed is a fine smooth stcel rod and a number of uniform 
elastic bands, one for cach point. | 


Going back to Gauss, one can say that he not only laid the the- 
oretical foundations for least squares but he also gave careful atten- 
tion to methods of calculation, and he put them into practice. He 
developed an abridged solution of the normal equations, and showed how 
to find the weights of functions of the adjusted values, and the sum 
of the weighted squares of the residuals, right along with the solu- 
tion. The Gauss solution is found in most texts; in particular Com- 
stock’s® should be mentioned. Doolittle's method, and many modifica- 
tions of it, are all based on Gauss' plan, which took full advantage 
of the symmetry of the normal equations, Naturally, modern elaetric 
calculating machines have brought about changes in routine here and 
there. 

In the present day, least squares can not be dissociated from 
the statistical researches that have been made since the writings of 
Gauss, Encke, and Helmert appeared, Most of today's students of 
statistics are familiar with what has been done in that line in our 


own generation, and no description of it need be made here; instead, 
some of it will be woven into the material that is to follow. 


ee ta a anc are eo ara scan mss scans on awe nen SS mah ont a at wn ees nb ten in SOR SOW Or Se SS 


* BR. Meldrum Stewart, Phil. Mag. (London) 40, 217-227, 1920. 

** Horace S. Uhler, J. Optical Society 7, 1043-1066, 1923, 
Corrado Gini, Metron (Rome) 1, No. 3, 63-82, 1921. 

++ rowell J. Reed, Metron 1, No. 3, 54-61, 1921. 


© George C. Comstock, Method of Least Squares, (Ginn, 1890). 


[Sec,. 1] 


Oa the part of many recént writers in statistics it can be said 
that an occasional lack of femiliarity with the memoirs of Gauss and his 
contemporaries has. caused no little confusion in the minds of students, 
More than once a profound. "discovery" in modern statistics has turned 
out to be something already known under other forms to students with 
Classical training. At the same time, many modern writers in least 
squares, especially of textbooks, have ignored what was being developed 
in statistical studies, and have, moreover, developed the subject of 
least squares from an incoherent perspective of the adjustment of obser- 
vations as it occurs under the varietics of conditions that may be 
imposed, All this has retarded progress, The pages that follow may 
possibly be a step--they can not be more--toward unifying some closely 
related topics in least squarcs and statistics. A knowledge of least 
squares is, to my mind, requisite even if one prefers to uss other 
methods of curve fitting, 


[Sec. 2] 
I. SOME SIMPLE PROBLEMS IN CURVE FITTING 


enc it ta OL Lena emnas: | perore S0iig into she 
general problem of the adjustment of observations by least squares, 
it will be helpful to look at some very simple applications in curve 
fitting. It is a fact that the simple ones afford nearly as much 
opportunity for thought in the field of statistical inference as the 


more complicated ones do. In all of them the principle of least 


squares requires the minimizing of x *, where 
emit i/o +) L wires? (1) 


the summation of the weighted squares of residuals to be taken over all 
observations, 0 being the r. m, s, error of observations of unit weight. 
This is the equivalent of Gauss' pronouncement quoted at the opening. 

In curve fitting, both x and y observations may be subject to error, 
and the summation of the weighted squares of residuals will then include 
both coordinates, and will Necuatenen ty be written explicitly as 

a WyVy" ). For an explanation of the symbols V, and Vy sce 

Fig. 9 on page 82. For convenience, the symbol 9 * will sometimes be 


used to designate the summation of the weighted squares of residuals, 


as in Eqs. 3, &, 30, and elsewhere; and we may then simply say that 
pe slg #/ge (1a) 


\ 


For an interpretation of x * see the exercises in section ©, 


[Sec, 2] 


Now since o is a constant for one particular problem, x * is 
a minimum when ¢* is a eeprarti hence we may think of least squares 
not only as the minimizing of y ® but also of ¢ = 21.,6.,,of the sum 
of the weighted squares of the residuals, Least squares may also be 
considered the minimizing of the estimate o(ext), as will be explained 
in, section 6c, 

Another way of looking at the problem is to say that the 
principle of least squares is the maximizing of ea el. and that we 
seek the solution that gives the greatest probability on the chi-test. 

The principle of least squares has been stated. Tho method 
of peeet equares is a rule or set of rules for going about the actual 


computation, 


[Sec. 3] 
5. Tg, SLRLESL GHAR, y. 
of curve fitting is the single 10 ° 


sample of n observations of equal 


weight on an unknown quantity a. 


ro) 


Here the curve to be fitted is the 
simplest of all curves 

x=a (2) 
containing just one "parameter", 
namely a, which is to be esti- 


mated by least squares, Here in 


oO 
SP ete Ee i ee em ee 


this simple case where the weights* 





are all unity, dd 32 35 34 Ge 


i} 


Pemeellidr) 2, res * 


Fig. 1. Ten observations 


tf 


Ge) Sia ales 


aL 


The y coordinates of the 
of equal precision (equal weight) 


points are merely the ordinal are made on an unknown magnitude 
a. The true points are connected 
numbers of the observations, and by the simple relation x = @, 
hence x = a is the curve to be 
hence are without error, so only fitted, The least squares value 
of a turns out to be x, the mean 
x-residuals appear in the summa- of the ten observations. x = @ 
is the "true curve", and x = x 
tion. Now x * is a minimum when is the "calculated curve", The 
calculated points are shown by 
the numerator the crosses; they all lie on the 


calculated curve. 


is a minimum, The observations, having once been made, can not be 


changed, hence the only variable in Eq. 4 is a, By giving various 


* For a discussion on weights see soction 5. 


soe 
[See, 3] 


values to a, 6* is made to take on various values, and the minimum 
will occur when the derivative* 
ag ®/ da = -2% (x, - a) (5) 
vanishes, that is to say, when | 
BCK 0 alee Oe orl: x) ne Se 
The least squares value of a is accordingly 
aie (2 /ny Ss x= x (6) 

So the vertical line x = x in Fig. 1 gives the smallest possible 
value to the sum of the squares of the residuals, and hence to XY os 
The line x = x is the "calculatsd curve"; it is the "curve" x = a 
fitted by least squares. On this line lie all the "calculated points", 
these being the least squares estimates of the observed points, In 
this very simple problem, the calculated points all have the same 
abscissa, namely x. (Compare Fig. 1 with Fig. 9, p. 83). 

The goodness of fit may be judgea by the -vaiue of ¥ 2 /(n-1) 
comparsd with o *, in other words by P(X) Pee n-l degrees of free- 
dom, This subject will be touched again in eink 6c and elsewhere. 

Note that the value of o is not regaiceee the application 
of least squares, since Sei Ces: x Sirti a minimum when 9% * is, 


o did not occur in Eq. 5. o is required, nevertheless, for the use of 


4 


the chi-test for the goodness of fit obtained, 
Note also that if s denotes the S. D. of the n measurements, 


then x * = ns */o ®, and the minimized value of 92 is ns®%, Fora 


* ds * is to be interpreted as the differential of ¢2, not as 
(a3)*°. dg */da is the derivative of 22 with respect to a, 


pa Ee 
(Sec. 3-4a] 
new sample of n observations there will be a new mean xX, a new line, 


and a new X *, 


4, The same problem with unequal weights. (a) The solution for 
PUL PLI SSL SOD PUPIL UVP SeaeaNmg RSS = NST ESE REPRE aa, 
the parameter a. Suppose that the n observations X,, Xe, «.. , X 
have weights w,, We, ... , Wy, perhaps not all equal; then 
Pra Tone wen cay ey (7) 
the i-th residual being, as before, x, - a. We are now to make 


i 
gets Sey etx, =a) (8) 
a minimum with respect to a. The derivative 


dg 2@/da = - 2 3,w, (x, - a) 


is to be set equal to zero, giving 


» Wa (x, ~ a)) = 0 
i.e. aliw, = 2 wx, (9) 
or 2 ieeay wx, / 2 i oe (10) 


where x is now the "weighted mean" of the n observations. In the 
event that w, = W. = ... = W this result reduces to the previous 
value of a in Eq. 6; in other words, the problem of the preteding 
section was a special case of this one, 

The minimized value of ¢* is here 


peeve Sows (xy 


he x)® = x wix" - > Ww, (11) 


em 
[Sec, 4b] 


(bo) The solution in tabular form, In sections 16 and 17 we shall 
s¢e a systematic procedure for the solution of normal equations and for 
calculating the "reciprocal matrix", in which are Aes the variance 
and product variance* coefficients; also we shall see the minimized 
value of 4* calculated right along with the solution of the normal 
equations. In simple problems like the one just considered, there is 
only one normal squation (Eq. 9) and it is of course very casily solved 
(see Eq. 19), Nevertheless it is interesting to see how the routine 
process that is to be learned later applies here; let us therefore set 
up the following tabulation, and perform the steps indicated, not 
worrying why. When more complicated problems are followed through, as 
in sections 16-20, it will be helpful to recall that the same procedure 


led to known results back here in simpler cases. 





No, a = 1 C 
if 2 Ww 2, WX i 
a ee We @) 


If the top row is divided through by 3, w (found under a) we find 
3 1 Swx/fw 1/%, w 
Interpreted-this means that 
a => wx/ Lw 
as already seen in Eq. 10. Moreover, in the C column of No. 3 we find 
1/= w, which can be considered the one and only term in the reciprocal 


-———— Se me a ee en ee ee es we ee ee ee ee a ae ae i a we ee eee es ee ee 


* I use the term product variance rather than covariance, following 
Dr. A. C, Aitken of Edinburgh, 


ilies 


matrix, that term being the reciprocal of the weight of a-- in other 
words, the variance coefficient of a, which is to say that 
ot) a) Pay arama rye. cw . | (12) 

This equation will be understood better after the discussion on weights 
has been read. (next section), 

Now let No. I be multiplied through by -2 wx/X w and added to 
No. 2; we find in the "1" column (the other columns are disregarded) 

wat > (2, wx) FP /da wy ore wx ta xi? Rew 

and this is none other than the minimum value of # =. which has come 
forth without the intermediate step of computing each residual and 
squaring it. These results will have a fresh significance after 
sections 16 ff have becn studied; see in particular exercise Se of 


section 17 (p.109), and applications in section 19, 


-12- 
[Sec. 5] 


D4 A digression to define weights, By definition, the weight 
Wr Of the function f is inversely proportional to the variance O* of 
f. That is to say, 1/wp is the variance ceefficient 7f f. In symbols, 
Wp = 02 Joe or ont = (1/we)o? (13) 
o* is simply a proprrtionality factor, and is evidently the variance of 
a function of unit weight. If o* be arbitrarily doubled, and Wp also 
doubled, of is unaffected in value, 
For example, let f be X, the mean of the n observations x,, 
Xe, +--+, Xn, Which are random variates taken from a universe of S. D. 
o, hence each of unit weight, Then, since the variance of x is o%/n, 
Eq. 13 tells us that 
wz = 08/( o@/n) = oriiceaa(lifae™ (14) 
whence we see that n is the weight, and 1/n the variance coefficient, of 
x. Or, if the n original observations were each of weight w instead of 
unity (as we could as well say, since weights are relative and not 
absolute, depending as they do on the arbitrary factor o*), then the 
variance of single observations would be o?/w, and the variance of ¥ 


would be one nth as much, so in this case Eq. 13 gives 


we = 0%/(o*/nw) = nw or o=® = (1/nw)o? (15) 


saying that nw is now the weight, and 1/nw the variance coefficient, 
of x. So, as before, the weight of X is just n times the weight of a 
Single »bservation, 


As E, B. Wilson* stated it: "The primal conception of a weight 


is that of a repeated observation." In Fisher's terminology, the 


OP ES me Sew SOS ED Bae ER kek O08 ES A mS SP Se ED SO BS AE A OO Se SE ele Se ee OD Ee Se OS am SS Sy Se com hte SEE Gin Ua SES OES OES GS Un age ce SB pee Gov och ems Ot Gus Ole oes Gm aoe Sad ee ae 


* £. B. Wilson and Ruth R, Puffer, "Least squares and laws >f popula- 
tion growth,” Proc, Amer. Acad, Arts and Sci. (Beston) 68, No. 9, 
August 1933, 
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[Sec. 5] 
mean x of n normal observations contains n times as much information 
as a Single observation. 

Concerning two functions f, and f,, it can be said at once from 


Eq. 13 that 
Wi 12 Woolies: 03% (16) 


which says that the weights of two functions are inversely proportional 


to their variances. 


Exercise 1. Given x* = (1/o#) 2 w V* as in Eq. 1, show that 


y? may be written v 


o//w 





pe Pe ey ram be 
that is, x® is the sum of the squares of the residuals, each residual 
being measured in units of the S.E. o//w of the corresponding observa- 
tion of weight w (compare with Eq. 20, next section). In other words, 
X* is the sum of the squares of the standardized residuals. X*sis 
therefore independent of the units used in measurement; a change from 
feet to inches or centimeters, or from pounds to ounces or grams, 
changes the residuals, but not the standardized residuals, nor ye. 

Exercise 2. When both x and y observations are subject to 
error, one may wish to designate the summation explicitly as 

X2 = (1/02) & (wv? + wyVy?) 

as has already been indicated in section 2. Show that this may be 


itt 
written Vx 


ae Ph les sr 


NG Rae SE 
(SFr, } 
which again says that xX is the sum of the squares of all the residuals, 


each one being measured in units of the S.E. of the corresponding obser- 


vation on the x or y coordinate (see the figure on page 83}. So xX® is, 


wit hm 
[Sec. 5] 
as before, the sum of the squares of the standardized residuals. The 


remarks in the preceding exercise hold. 


Exercise 3, 6*, or the sum of the weighted squares of the 
residuals is, like y *, also invariant to changes in units (as from 
pounds to ounces, stc.). But g * is dependent on the arbitrary choice 
of 0, whereas ¥ * is not. One weight in the whole set is arbitrary, 
and the others are related to it through Eq, 13; fixing this one 
weight is cquivalent to fixing o. g * can be doubled by doubling all 
the weights, but this has no effect on x * because o * would also be 
doubled. The least squares solution for a (and other parameters, if 
any, as in more complicated problems) is independent of o *; the 
parameter or parameters that minimize ¢* for one set of weights will 


also minimizg it if all the weights are doubled. 


Note: For another interpretation of g* in curve fitting, see Ex, 3 


of section 15, page 97. 


For other exercises in weights, see page 30, 


ai Be 


is 
6, Several samples, all on the same unknown, 
VELAPI PAL LILVISPOI PIV. 
tions have the sams precision, 


UV LOU SU SUT UV 


(a) All observa- 
Let us suppose that n observations of 
equal weight (equal precision) and all on the same unknown, 4s for 


example those of section 3, are arbitrarily subdivided into m samples 
Bee Mig Bip vee » Ty observetions. 


We shall say that 
Xy 


is the mean of n, single observetions 
Xe vy " 


w 


tt 


rt 


x ne rt v had n La wv 
m m 
Now if 


single observations have unit weight, then it will follow 
Wy 


from Eqs. 14 or 15 that the weights of the m means are 

= iy, zl Me, 
We may now consider the m sample means to be m observations of weights 
respectively ni, Ne, | 


ss. bs Digs 


> wk 
X==—— 
ZW 


to which the results of section 4 apply. 
The value of a thet minimizes x ® is then by Eq. 10, 


Tahy TF Wyte 7 iss. 


net 





(17) 
This X is the "weighted mean" of the m samples, it is simply the 
arithmetic mean of toe n,' * Wat save" D 


L 


2 
her 
\ 


_ single observations since they 
all are of equal weight. Deviations (V.) reckoned from X make 
mai (1/o 2) Iw é 
a minimum, and it is noteworthy that, at the same time, XV = 0. 
It should be kept in mind that X is independent of just how the 


n observations are subdivided, but X 2.45 not. Moreover, both X and x * 
will vary from one set of observations to another. 


oper 
[Sec. 6a] 


ce 


~a~ f= 
Ng ,Sg 


Fig. 2. Three Series of Observations on a Magnitude u 


n, observations have mean X, and S, D. s,. 


n 2 t Al ? Xs " v s Avs 


X is the weighted mean of the three series, 


The errors and residuals in the individual means X,, X,, Xz 
are denoted by E,, E,, Ez and V,, Ve, Vz respectively. The 
error in the general mean X is denoted by E. As the figure 
happens to be drawn, E, #,, Ez, Vi, and Vg are positive and 
E, and Vz are negative, as the’ arrows indicate, This case 

of curve fitting is intermediate between the simplest prob- 


lem shown on page 7 ahd the more general one on pages 82 
and 33. 
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6, Several samples, all on the same unknown, (a) All observa- 

PANNA Annan UE UE SUI OL er OT ale Sed 7 

tions have the same precision. Let us suppose that n observations of 
equal weight (equal precision) and all on the same unknown, as for 


example those of section 3, are arbitrarily subdivided into m samples 


Of Ni, My, «ee » Dp observetions. We shall say that 


X, is the mean of n, single observetions 


Vii hatha rr AD hay tt tt 

° 5 

xX ae a nT ron " tt 
m m : 


Now if single observations have unit weight, then it will follow 
from Eqs. 14 or 15 that the weights of the m means are 
w= Ty We Foes eee oe 7) Op 
We may now consider the m sample means to be m observations of weights 


Pecmwecrively Ni, Ng, ss. 5 to which the results of section 4 apply. 


m? 


The value of a-that minimizes XY ® is then by Eq. 10, 


> wk u Mia Rall te Te ay feet to 





(17) 


This Xx is the "weighted mean” of the m samples; it is simply the 
arithmetic mean of ae ge od Feet A single observations since they 
all are of equal weight. Ppeeiat tous (Vv, ) reckoned from X make 

= {1/o'“) ow -* 
a minimum, and it is noteworthy that, at the same time, 2{V = 

It should be kept in mind HOAs og 8 Wenres 060 of just how the 
n observations are subdivided, ‘put x ® is not. Moreover, both X and X & 


will vary from one set of observations to another. 


ae 
[Seec, 


Ng ,Sz 


5 FS — Fe ee 


a ae 


Fig. 2. Three Series of Observations on a Magnitude u 


n, observations have mean X, and S. D. s,, 


X is the weighted mean of the three series, 


The errors and residuals in the individual means X,, X,, X, 
are denoted by E,, Eg, Ez and V,, Vz, Vz respectively. The 
error in the general mean X is denoted by E, As the figure 
happens to be drawn, E, E,, E3, V,, and Vz are positive and 
E, and Vz are negative, as the arrows indicate, This case 
of curve fitting is intermediate between the simplest prob- 
lem shown on page 7 ahd the more general one on pages 82 
and 33. 


6a] 
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[Sec. 6b] 

(b) The precisions of the single observations differ from one 
sample to another. Suppose that 

X, is the mean of n, observations from a population of 
$.D.0,, the variance of X, being o2/n. 

Xn is the mean of Qn observations from a population of 
S.D.0m, the variance of X, being Se bike 
Sis Os, «++ » Up need not all b3 cqual. Then we may take 


w, = 0%/(of/n,) = n,0*/o2 for the weight of X, 


hee 0-7 (0c/n2) = n.d7/og) wet 7 eae eee Cis) 
nee eo (on/n.) = no e/o ue ttm : eo 


of is arbitrary, i.e. the weights are purely relative, Eq. 10 
applied to this problem gives 


>wX make /Gi + 02k, /o8 + 0 + on / Off 


Ewan Hoyo in, /oP + eee / 02 (19) 








for the least squares value of a. This X is the weighted mean of the 


m sanples; deviations (V;) reckoned from it make 
nye = (1/0 =)inv é (as defined in Eq. 1) ae 
a minimum (see Fig. 2). At the same time, these residuals give >wV = 0. 


Note that the problem of part (b) reduces to that of part (a) 


Se On ee Oye S64 ot) On’ 
Note that o does not appear fnetne traction of fq. 19, 1.6.54 

is independent of 0. If o% be doubled, all weights would be doubled, 

but X would be unaltered, Likewise y¥* in Eq. 20 would be unaltered. 


(See the exercises in section 5). 


eta 
[Sec.6b,c] 


Note also that x* can be written 


n (Discrepancy between X; and X)* 
c< = nn rn cS EE 
X Variance of Ay about true mean 


see Ex, 1 of section 5, page 13, 


(c) The estimates of o. On account of the distribution*of y* 
when the actual sampling (the experimental work) is described by the 
mathematical model here assumed, namely, normally distributed observa- 
tions, the mean value of X* in the long run is equal to k, the number 
of independent residuals or "degrees of fresdom".** For the problems 
of parts (a) and (b), this number is m-l1 on account of the fact that 
there is one relation (Eqs. 17 or 18) between the m residuals and X. 


The unbiased” estimate of o% made by external consistency’ is found by 
* Karl Pearson, Phil. Mag. 50, 157-175, 1900. <A’ paper dealing more 
specifically with curve fitting of the kind here considered will be 
found in the J. Amer, Stat. Assoc. 29, 372-382, 1934; see also Phil, 
Mag. 19, 389-402, 1935, ti 


** The correction for the number of unknowns evaluated (one in this 
case), and the equivalent of setting the mean value of X* equal to g- 
divided by the number of observed quantities diminished by the number 
of unknowns evaluated, were set forth by Gauss in his Theoria Combina- 


eee 


tionis Observationum Erroribus Minimis Obnoxiae, Pars posterior 


rr ee a ee 





(Géttingen, 1823: vol. 4 of his Werke) Art. 38. This correction is 
sometimes: credited to Bessel, though in my own writings I have. cited 
Encke; but the reference jyst given, for which I must thank my mentor 
Dr. G J. Lidstone, places the originality with Gauss. 





* Unbiased in the sense that its mean value is 0%; sco J. Neyman 
Washington Locturos (The Graduate School, 1938: $1.25) pp. 131, 132, 
tS 5 rah) |e 





++ The terms external and internal consistency were introduced by Birge 
(Physical Review 40, 207-227, 1932) in a discussion on the mean value of 
the ratio of these two estimates, the occasion being a paper by Scarbor- 
ough (Proc. Nat. Acad. Sci. 15, 665-668, 1929) denying the validity of 
the extcrnal estimate. The comparison of the two estimates (p. 21) is 
an application of the "analysis of variance", the essential features of 
which have long been recognized by physical scientists; sed for example 
A. de Forest Palmer Theory of Measurements (McGraw-Hill, 1912) pp. 66-71. 





Bs i 
[Sec. 6c] 
arbitrarily saying thet x* has its mean value k, in other words that o 


satisfies Eq. la, page 5, whence comes the estimate 
o2(ext) = g@/k (21) 
From this we get, for the problem of section 6b, 


Miesiarr kl | Y 


o2(ext) = (Sw? ) / = : (22) 


m-1l1 

This estimate is made from the external consistency of the data, 
i.e., from the fit of the "curve" X = a, What we do in making the 
estimate o(ext) is to say arbitrarily that y¥* does equal k. This is 
equivalent to saying that P(x) -is about $--not exactly s because of 
the skewness of the y® distribution, which, however, gradually disap- 
pears with increasing k. 

If we are not positive that all m samples came from populations 
having coincident means, we should have as an alternate hypothesis that 
the m population means ,, Ue, ... , Up, are not all identical, Now 
if one or more of them really are not equal to the others, o*(ext) is 
raised, on the average, to some value higher than o*; consequently in 
ested in knowing if o*(ext) is significantly greater than o*, or, what 
is the same thing, if x2 is significantly higher than k, This can be 
ascertained by looking up P(X) in tables of chi-square. Of course, 

X* can not be computed nor compared with k unless o is know. Dei sha) 
use Fisher's tables of: z, one would set 


in [o#(ext) /o® } rien Es 


dole 


By ae 
and look up P(z) with Fisher's n, asm -1, and with ng, equal to 


infinity, since o* is here assumed known, 


(Sec. 6e] 
Now the 
Wt. of X = wy = in,0%/o# oe on be 
whence the 
\Webiais, er or ae o (ext) /wy 
= SwV* /(m-1) yee o /(m-1)wy (25) 


wherein wy is to have the value given in iq. 24, 


There is also the estimate of o made from the internal consistency * 


of the data, i.e. from the consistency of the observations within samples. ** 
This is* 

tO in oh, 3) eee oe (26) 
whence the 

(Est'd S.E. of X)¥ any = 0? int) /wy (27) 


wherein wy has the value given in Hq. 24, 


ee eee ee ee ee ee ee ee ee ee ee ee Be ee eR EO Be ee RE Be ee ee ee 


* See the reference to Birge on‘page 18, 


** Tt is assumed that there is more than one observation in each sample, 
whereupon the estimate by internal consistency is actually possible. 
Unfortunately as data are too often taken, there may be but one observa- 
tion at each point, end the estimate of o by internal consistency is then 
not a shawn This is to be regarded asa technical fault of 
experimentation, a matter of design, 

* See, for example, Eq. 67 in Deming and Birge's Statistical theory 
of Errors (obtainable from The Graduate School, 35 cents) p. 158. 


im 
[Sec. 6a] 
(a) Comparison of the two estimates, As was mentioned in part 
(c), the estimate o{ext) is valid only if the m populations have coin- 
cident means; if any two of the means Ui, Ue, ... ,» Uy, are unequal, 
o*(ext) is, on the average, raised above o* . But, in contrast, the 
estimate o(int) is unaffected by inequalities among the means of the 
populations; so long as o remains the constant S.D. of all of them, 
the average value of o*(int) is still o* It follows that a statistical 
test of the hypothesis U, =U, = ... = Up, is to examine the ratio of 
the two estimates. To do this, we may follow Fisher and take 
z = 4 1n (o®(ext)/o®(int)] (28) 
and look in Fisher's tables to ses if z is significantly different 
from 0. (In doing this, we use m-l for Fisher's n,, and n, + ng + ... 
+n, ~ m for his ng). If z is found to lie beyond the 1 percent :. 
limit, we say there is "statistical evidence" that the data are not 
homogeneous, or-that not all the uy are equal; in other words, that. the 


curve : 
=e (29) 


is not a good fit. Such a calculation takes account only of the! 
numerical data, and any conclusions therefrom must of course not be 
taken too seriously, but rather should be weighed along with the 
considerations of the experimental work, and previous experience. 
The student will realize that a Peet aeh of two estimates 


of the same o is an “analysis of variance" (cf. footnote on page 18). 





R. A. Fisher's Statistical Methods for Research Workers (Oliver & 
Boyd) contains many examples, & particularly good one being Example 


42, "Test of straightness of regression line”. 


-2?2= 
[Sec.6a-7a] 
It must be kept in mind that a now set of data will give new 
values of o(ext) and o(int), hence new values of P(X) and ‘P(z). Grand 
is these variations that statistical tests lay odds on, the assumption 
being that the experiment is "under control"; see Lecture II of Neyman's 
Washington Lectures (The Graduate School, 1938); also Shewhart's lectures 


(The Graduate School, 1938), 


ten sain Rinne e of a lines that is known to 


VLSI PLL DILL BLU UU UTI 


pass through the Origin. iIn general ae section 1 we may say that 


WVU 


dite Seer VE tue) 


So far, we have seen only 
special cases of this-~spccial 
in the sense that the error 
was all in the x coordinates 


of the points. Now we take 





(a) The y coordinates 


subject to error; x free of 


Fig. 3. A line known to pass 
through the origin, the slope 
to be estimated from the ob- 
served points. 


error. Li Wy ule ts the 
weight of ¥3 then 

g* = Iw. iyy7be)? | (30) 
is to be minima yy is observed, and bx, is its PR binge value; 
the Mererence force then. is ee Vertical ory Beat cued. at the 
i-th able Wo differentiate 62 end ‘obtain 


ag2/ab = ~ 220; x; ae 


i 7 dx4) 
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[Sec. 7a] 
Set equal to zero, this gives | 
2 = . 
Dow XP = LWyxXyVy . a 
whence 
b = Swry/Lwx* AoE 


Note that here, 2, w res # 0, but that 2 W.x.res = 0. -How does 
this come about? (Cf. remark 3, p. 126 for further reference). 
Special case: Wy proportional to ify! tie. the 8.5. of v4 is 
Peycrtional to Xi. Then Eq. 32 gives 
b= X y/rx ; Ay (33) 
Note that here, 2, w.res = 0, 
Another special eee All weights equal, oe. all Yi have the 
same S.E. Then iq. 32 ives. | ; 
ery at | (34) 
This is perhaps @ more usual one than the previous special case, 
Note carefully the distinction between Eqs, 35 and 34, 
Ses how in Eqs. 32 and 34 a point has more influence if it is far out- 
lying, the influence ot Rrcaey oe ee: to x. Is this réasonable? 
But in Eq. 33 all points ars equal in influence, near or far. Why 
Paevid’ this be? or rather, under what conditions would Hq.-33 be a 


reasonable result? 


(Sec, 7b] 


(b) The tabular calculation of b and its weight. This will be 
similar to the tabulation in section 4b (q. v.). 
Suppose we write 


No. db = 1 C 





r Swx® Swxy 1 
2 ee 2wy * O 
S fee _(2wxy )* ee (Gotten by multiplying I 
OI» dice through by -Lwxy/Dwx®) - 
II ee Swy? ares A igh (By adding 2 and 3) 
Ve 


ee i a em a a a a a ew we ee ee ee 


No. I solved with the "1" column gives b as in Eq. 32. 


of Se ” Y iy ee C “Eve v pes 1/Swx® , which means that 


re Swx 2 (35) 


No. II gives the minimized value of ¢* or of Xw, (y, -bx, )* directly 
(found Tethe "1" column), without the intermediate step of calculating b 
and the individual residuals and their squares. The student should 
follow this nia especially after studying section 17ff. 


Here we have 


o Xext) = g® / (m-1) (36) 
and the 
d g® OF eae 
(Est'd, S.E, of b)® 44 = (m-l)w, > (m-1) iw? (37) 


In order to apply the t test to see if there is "statistigal 
evidence" that the calculated value of bis significantly different from 
some theoretical value, say B, we should write 


|B-b| 


t | (38) 
Est'd S.E. of b 
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[See, 7b,c] 


and make the t test with Fisher's n equal to our,m-l, The region of 
rejection in the t distribution is to be chosen with duc regard: to 
admissible alternative slopes, which may be greater or less than B. 
In the denominator of Eq. 38 we may use the estimate made by external 
consistency, or that made by internal consistency (section be}, LE 


o(int) were used in place of o(ext) in Eq. 37, then we should have 


fMetias.n fof 6/40. = o®(int)/m, = o%(int) /owx® Dundee) 


This would replace the denominator of Eq. 38, and the number of degrees 


of freedom (Fisher's n) would be the total number of observations di- 


minished by the number m, 


The. testing of parameters is closely tied up with the theory of 


confidence intervals for any function of the parameters; see page 116. 


(c) Now let the x coordinates be subject to error, y free of error. 
w, will now denote the weight of x;. In place of kq. 30 we now have 
g® = Lw,(x,-y,/d)? (40) 
since here the y residuals are zero, and ~* is made up by squaring the 
x residuals, By differentiation 
ag2/ab = +(2/b®)Zw,y4 (x, -y,/0) (41) 
Set equal to zero this gives 


b © wey = 5 wy®, or b = &, wy®/X. wxy (42) 


Note the distinction between Eq. 32 and 42. 


Ye 
[See, 7c] 


Exercise ii: If b in Eq. 42 be distinguished as b', prove that between 


Kas. 32 and 42 there exists the relation 
‘b/bt = r2 (43) 
r being the correlation between x and y. Hence b = bd! only if the 


points all lie exactly on a line, since then r® = 1 and b =-b’. 


Exercise 2, Prove by making up a tabulation for b' similar to the one 
on page 24 for b, or by any other method, that the weight of b' (the b | 
in Eq. 42) is 2% wxy; in symbols 


Wpt = 2, wxy ) (44) 


Exercise 3. Prove by the same tabulation or otherwise that in section 


7c the minimized #4* -is 


B2 = yi wx® - (L wxy)? /X wy® (45) 


[Sec, 8] 


8. The propagation of orror. Let F be a function of x, y, 2. 
Then if x, y, z are in error by Ax, Ay, Az, F will be in error by the 
amount AF, where by Taylor's series 


AF 


" 


Ay + F,a2 + oe. (46) 


PLAX 7; Fy 


wherein 
i dF dF dF 
Py = ae Fy = ay P & 
Formula 46 is often callcd the propagation of error, 
Now square each side of Ea. 46 and get 
CE ( Fax) + (Fy Ay)* + (FAz)* + 2B FP Axby 
{ + aF FL AxAZ + eP Pe Ay bz Fa eats 


Imagine a F ae to be nearly constant as Ax, Ay, and Az take on all 


alae 
possible values.* Then let each term be replaced by its average value; 


the result is 


Bivlvs 2 z 2 
oF (F,9,) + (Boy) + (Eto. ) + 2( FF yO, yl yy 
+ FYF20,0zrx2 + FyF20y%zTyz) N ighenetieenenibe (A? ) 
where OF = variance of x, Pay the correlation between x and y, etc 


This formula (also the simplified form in Hq. 48 when it applies) i- 


called the propagation of mean square error, or the propagation of 


variance. 


The terms in parenthesis are zero if the errors Lhox. Yolande 
are independent, i. e. uncorrelated. In such a situation Ey. 47 re- 


duces to 


_ _- — _ _ — _ _- =— _- _ _ _ _ _ -_-_ — _-_ = - _— _ _ a —_ — - _ _ - _ _ -_ 


* Ir practice it may safely be assumed that the ranges of variation in 
Ax, Ay, and Az are not large, wherefore the constancy of Fy, Fy, and Fz 
is usually not a difficulty. It is moreover presumed that o,*, OF, 
and eae actually do exist (as would nct be the case, for instance, in 
a Cauchy distribution of the errors in x, J, and 2). 


[See. 8] 


It is interesting to see that if F be taken as the mean (x) of 
the n independent sbservations X,, Xe, ..- 4 xX, each of S. HE, o, then 
Eq. 48 leads to the well known expression 


o;* = 9 2/n 


‘e 


as was taken for granted on page 12. This, however, does not tell us 
that if the individual observations are normally distributed, the mean 
x is also--this fact must be obtained otherwise. Eqs. 47 and 48 are 


in fact independent of any assumption concerning the distributions of 
the errors in x, y, z, and F, provided.the S. Es. o,* etc. actually 
exist, as was stipulated in the footnote on the preceding page. 


Exercise 1, The mean square error of the sum or difference of 
» 
two numbers having equal precisions is twice the mean square error of 
either alone (assumed independent). The r.m.s. error (S.E.) of the sum 


or difference is V2 times that of either alone. 


Exercise 2. If uis a linear function of the independent variables 
xo -¥. and Zz, say 
Na AP: Yai oy. +o 
then the r.m.s, errors are rolated by the equation 
Na = a*ofg + b¥0f + c®of 


A special case is exercise 1, 


Exercise 3, If u = axyz, then 
ue{(o,/x)® + (oy/y)® + (o,/z)*} 


or CA (o/ at (oy/y)* te io alee 


tl 


o* 
u 


which interpreted says that the squares of the percentage errors are . 


additive. 
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Exercise 4, If wu = axy/vz, then 
ee e/a) en osei se (o/eat (07 /2)" 
the squares of the percentage errors being again additive. 
Myercise 5..+1f u;> ax? y® zy» then 
(oy/u)®= (a0x/x)* + (Boy/y)® + (yoz/2)? 
Here the percentage errors are increased by the factors Ct ayes 


(Exercises 3 and 4 are special cases of this). 


Exorcise 6. If A = nv® (A the area and r the radius of a...- 
circle), then if an error Ar be committed in measuring r, the corres~ 


ponding error in the area is closely 


AA = 2nr Ar = 2(A/r) ar 
AA a or 
A eit 


An error of 1 percent in the radius means about 2 percent error in the 
area. Also 
On = 20, and w, = (1/4), 


(special case of Ex. 5). 


Exercise 7. For the conditions of exercises 4 and 6 the 


—-— 


relations between the weights are respectively 


i 


i/ Wh 1/x?w, + 1/yPwy + 1/2*w, 


and. 


tween - 


<6 pias ' - . or ne? - ne j Z } ye he 
» l/u Wy & ew, + BE /y* Wy ‘ ¥ [2 wy 
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Exercise 8, (a) Let* Y= 1n y,-then 


= 
Ke 
TT 
eq 
st 
= 


' (This result is very important; see exercise 18 in section 19) 


(b) If y is in error by the amount Sy, log y is in error. (..4343 Sy) /y. 


(c) “It Y =sloe vy. “then 


Cue ra) 
oy = (0.4343 o,/y) 
A = 5 2 f 
Wy ey Wy 
Exercise 9. Lét u_= eine then 
of = bruv ok 


Exercise 10, The period of a simple pendulum is T = 2nv(L/g). Show 
that if the length L is too long by one-tenth of a percent, the clock 


will lose about 44 seconds per day. 


Exercise 11. (a) Prove that if F is a function of x, and x @ funcises 
of t, that 
FP Wy = PPL, 


where F, denotes aF/dx, and F, denotes ar/at. 


(b) Prove that 
1/w, = (dx/at)® /w, 


(These results are useful in section 19). 


——— ee ee nr a ee ee i es ee ws a ee ie ee 


* The abbreviation ln is used here for -"logarithme naturel," as is 
common in Europe and among chemists everywhere. The abbreviation log 
will be used for logarithms to base 10. 





puntos bir 
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[Sec. 9] 
II, MORE COMPLICATED PROBLEMS 
9, The general problem in least squares*. The fundamental 
ELMLIMLMSMLEN LISI LISSEALL IE MIMSSMIIASLIIIIIMIPLPIILPIII. LUV 


notions involved are the true curve (unknown), the attend points, 


and the calculated curve. We shall deal with the following quantities. 


Mmreerved coordinates**: X,., Xs, ...", Xny Xne2>-AXnee, ++, Xen 
Weights: Witt Wel acne, West We oy Wig es) ele Won 
True values (unknown): A Meee, OE eres, Chee ee. Se ORh 
Calculated coordinates: . X41, Xa, «+. » Xn3 “Xntas Xtra. e+ » Xen 


Residuals (Obs.-Calc.): Vi, Va, «+. » Vn3 Vnesas Vntos «e+ » Van 


In dealing Poe fitting, An+4 will take the ieee ot. Ya 
Xn+e the place of Y2, etc., foe conreniencs in writing summations 
(see Eq. 49 for Deere nea ie If the problem is not one of curve fitting, 
as for example if it were one involving geometrical conditions with no 
adjustable parameters, we need not think of ae observations as being 
Per instes of points, and in such problems the quantities listed above 


(X, w, x) take subscripts from 1 to n and no further, the subscripts 


aa 


n+l, nt+2, ... , en not then being used. 


re The development from here on is an amplification of three wae 


that appeared in the London Philosophical Magazine. The reference 
See vot. 11; pp. 146-158, 1931; vol. 17, pp. 804-829, 1934; vol. io 
pp. 389-402, 1935. 


** Ths use of the capital letters here is consistent with section 6, 
Since the numbers listed as observed coordinates may actually be che 
means of several observations. This is taken care of in the weights. 


[See. 9] 


In any case, the principle of least squares requires that* 


g® = LwiV% = Iw, # (49) 


0 
shall be made a minimum with respect to xy "ay Ae Se Butethis Te nee 

a Simple problem in maximum and nium rele the adjusted values xj 

are peveten to one another, ©.g. in the case of measurements or the 

three angles of a plane triangle, we should requires’ that x, + Xp + Xz = 180° 
(see section lla). What is more, in curve fitting, the conditions on 

the adjusted values xj involve unknown parameters a, B, y whose estimates 
a, b, c are. sought. In the. problem of, section 3, for instance, the ad- 
justed values of the x coordinates of the n points were all required to 

be sequal to a, which was then evaluated as x (Fig. 1) to make the sum 

of the squares of the residuals a minimum, So to take care of the 

general case we shall supposs that the adjusted values xX; are subject 


to v conditions, designated by 





Bs 


! . 7 
Eee Cay Xe, ees ’ Xen; a, b, c) = 0 | 
Py te bt y | v equations 
me ee? ) v > for . (50) 
| a v conditions 
Fiat )=0 


’ . < 


Diffcrent sorts of problems are distinguished by the different kinds of 


conditions that the adjusted quantities x,, and a, b, ¢ are subjected to, 


They are all convenicntly handled as one problem, because there is only one 


principle of least squares--the minimizing of y®*. 


ee A ee 1 eR te © ee en a RENT 


ee mee me me ee ee eee ee eee ee eee ee me ee eee ee ae ee a ee ee ee ee ee ee 


* The sign >}; will denote summation:with i running over all observa- 


Long. 
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These will be referred to as the conditions--the conditions 
imposed upon the adjusted values x; and the estimated parameters a, 
b, c (if any)... The assumption behind the whole process is that 
the conditions would all be satisfied exactly by the true (unknown) 


quantities being measured, and the true parameters a, BP, y. 


Let derivatives of Fo be denoted by subscripts*® as in Eqs. 46-48; 


Fo = ap/ax,, FD = ar™/ax,, ote. 
( Oda] 
pl = ar'/aa, FE = ar®/av, ote. 


wherein, after differcntiation, the observations are to be inserted, 


along with the best available approximate values ao, bo; Co (infra). 


iv , 
In other words, F, is to be a number representing our best guess ** 


at the value of the derivative ar? /ax,, and similarly for the other 
derivatives in Eas. 51. 


o . + 3 
We also writ? 


os en <> Oe eee OS ee oe we we es we we a en a ee we we ww ww i ew ewe ewww ee wwe ener ae eee error eee 


* Denoting differentiations by subscripts is a common practice in 
Mathematics. The superscript h, which may be 1, or @, ..., OF V, 
merely distinguishes onze F (i.e. ons condition) from another, 


** Tf our best guess .is too far off, a second adjustment may be re- 
quired, but this rarely happens in practice. Ses the quotation from 
Gauss in Ex. 4 of section 19, pp. 124-125. 


4 Mr. Jesse H. Buell of the Forest Service has kindly directed my 
attention to an inconsistency in notation; the subscripts 1, 2, 3, ... , 
a, b, ¢ in Zqs. 51 denote differentiations, but the subscript o in 
Eq. 52 docs not. I fear it will have to stand, Fortunately, confusion 
does not ordinarily arise, especially after the inconsistency is recog- 
nized, . 


BL ae 
[Sec. 9] 


Fo is a small number, the amount by which the condition Ei=.0 (Riis 
to be satisfied by the observed values X,, Xs, ... and the approxima- 
tions € 5, Do, Co. Similar statements hold for Fs, FB, ...', rieae 


@o, Do, Co are approximate values of a, b, c.: They can usually 


be arrived at somehow, as by putting three of the ra equal tolzere alae 


(method of.selected points*), The final adjusted values a,. b, ¢ will 
be independent of the approximations a 5, bo, Co, except that occasionally 
be strike tnk tare ie must not be too rough; see section 13, Each ma 
would be zero except for errors of observation and the consequent 
impossibility of choosing a,, bo, ¢,g to satisfy all the conditions 
exactly. By definition; Va = ag “ap wy cl, In piveneicouS Co. 

Now let the conditions be made linear in the residuals V,,; Vz, ... 
Ven; Vas Vy, Ve, by expanding Eqs, 50, retaining only the first powers 


of the residuals**, and rememberinz that Xq Ry eee Pee Bi ag S48 
The "method of selected points" could be considered a method of 
curve fitting were it not about the worst concsivable method,. and 
hardly worthy of the name, It throws away all the information con- 
tained in the points not selectcd; yet this much can be said, if there 
were no errors in any of the points, it would yield correct results, . 
Generally speaking, it is well to select the points as far apart as 
possible when adopting the plan. There are of course other methods 

of finding approximate values: there are eraphical methods, ‘the method 
of averages (Norman Campbell's method of zero-sum, Phil. Mag. 39, 
177-194, 1920), and Cauchy's method Comptcs rendus 20, 650, 1847, 
Frequently one will have good enough approximations from previous 
expericnce, When I speak of approximations by any of these methods, 

I do not imply that they are inferior to least squares--they are only 
different, The values ag, bo, Cg that any one of them gives may be 
very close to the least squares values a, b, ¢, and thus be useful 

in ‘the jieast squares solution, when such ssems desirable. ahaha 


** The problem of a straight line with no error at all in one of the , ne 
coordinates (Exercises 1 and 7 in section 19) is one .in-which there. 
are no squares and higher powers of the residuals to neglect, hence 

no discrepancies of the kind mentioned. The simple example of the 
triangle in section 11 will be another. On rare occasions the residuals. 
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The result is 
7 ie.) eee ay 
i v equations 


‘urov, + Pov, + Fin, + Fav, =F (55) 


These are called the reduced conditions, They are equivalent to 
Eqs. 50, except for small discrepancies arising from the neglect of 
higher powers of the residuals in the expansion. 


Now if 4® is at its minimum value, and if any or all of the 


residuals then undergo small variations (expressed by 8), the varia- 


tion in $2 will be zero-to within higher powers of the variations in 


the residuals; in other words 
=, 8g = dw.V;/8V,/= 0, one equation ' (54) 


The variations 5V,, 8Vz2, ete. are not arbitrary but must always 
permit V,, Ve, ete. to satisfy the condition Eqs. 50 or their equiva- 
lent (53). So by’ differentiating Eqs. 53 we find that 


h 


any 8V5 ce Fa h = Avy ae i 0 8 SN (55) 


h h 
Sy eet Pe avy, th Aeoy, =HO 
Ma are} a he vy equations 


Now multiply Eq. 55 thru by -Ay, an arbitrary multiplier; 


PPL OV) “MaFe oe : MP Svaredats 8¥o.= 0, be: Gem vee 
Add Eqs. 54 and 56 and collect cosffieisnts of the variations 6: 
; h eee oD 
Z(w,V, - Cyrp)) vy - (AgPg] dvq - [Fpl 5%, 
~ (AFR a5 =. 0 | one an eeean, ey ah 


may be so large that the neglected térms invedidate: the reduced con- 
ditions (53), in which event,.in general, no systematic solution is 

available. An exception is the straight line under. certain circum- 

stances of weighting; sée Exercise 6:of section 19. 
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In Eq. 57 there are two kinés of summations--there is the summation » 
in'which i runs over all observations; and there is also the summation 
over h, in which h runs from 1 to v,:i.@. over all conditions. The 


latter summation will be denoted by Gauss’ [ ]. 


This equation contains 2n+3 variations*, &V,, 65V2, 6Vs, ... , 
SVen, 5Vg, 5Vp, SV. But only entS-v of them are arbitrary on account 
of Eqs, 55, which are v in number. Let Ay, Ag, ..- , Ay be SO chosen 
that v of the cosfficients in Eq. 57 vanish; then the coefficisnts of 
the variations in the remaining en+3-v terms must also vanish, because 
they are used with an equal numbsr of variations, cach of which is 


arbitrary**. Then all the cosfficients in Eq. 57 vanish, that is, 


Ae h , : 
Vi = a, nF J. gn equations, i= 1, 2, ... , 2m (58) 


CAQEBT(S O,m Cyst. og Malta ial sao (59 a,b,c) 


The v Lagrange multipliers (A) are no longer arbitrary, having 
been chosen so as to cause y of the ecosificicnts Inviq. 57 to Vania f 


me ee a me eee we we ee ee a ae a eee ae ae a ae ea ae a a we ee ee 


* Here the number of paramstérs is taken as %.- If there were p . ; 


parameters, there would be 2gntp variations. For practice, the student 
should write out Eqs. 54-61 with (e.g.) n=3 and v=2 with (say) 2 

parameters. There is no other way to gain familiarity with the devel- 
opment. a, 


* This is the method of Lagrange multiplisrs; see his Mecanique . 
Analytique (1811) tome 1, p. 74; or Benjamin Williamson Differential i 
Calculus (Longmans, 1893) Ch. 11. The least squares problem without | 
paramsters was worked out by Gauss. He called his multipliers cor- 
relata, not mentioning Lagrange. Many texts in least squares use ise -the 
term: "correlates" or “correlatives" in this connection, but none that 
I have seen makes any mention of Lagrange, The refsrence to Gauss is 
his Supplementum Theoriac Combinationis Observationum um _srroribus 
Minimus Obnoxiae (Géttingen,, “1826; Werke , VWolLwwhpartcalls 








oe 


























nay = 


(vide supra). Their values are given by Eqs. 61. 
Now substitute — (NPE) for V, in the reduced conditions (53). 
if 


Collect the coefficients of Ay, Ae, «2+ » Ays Vas Vps Ves and in so 





doing set 
Irqs 1Tps al aS 
i = Fsks Forks + a PonFen ne i: (60) 
rs és TEs eee eee sr 
ws 2 Wen’ 


The following system of equations results. I call them the 
"general normal equations", For convenience only the co:fficients are 
tabled, the unknown being written across the top. On the left of the 
equality sign, each coefficient is to be multiplied by the unknown 
appearing above it, the plus sign betwccen terms being understood. On 
the right, each Fp is multiplied by unity, hence the heading AB ao Pop 


that column, 


The general normal equations 














Mg Ne Ag toe Nay Va Vp Vo ae it 
Lai Lei L31 eer Lyi Fs : Fp ‘os Fs iy 
Lie Lee Lge sae Lie 1s FS 2. FS | 
Lis Lez L33 eee Lys os FA = is (61) 
; ; ; : ; ‘ omy iit 

v ee _v Vv 
Liy Ley Lsv AP Lyy Ba tb re Bo 
oe ge Peed! tae f 0. 0 0 0 (59a) 
Boas Fy Elo ky Se pee a 0 Qiu ed (59) 
Fo Fo Fo ve FY 0 0 0 0 (59¢) 


-38- 
[See. 9] 


Note that since Lys = L the coefficients below the unknowns 


sr? 
are symmetrical about the main diagonal. 

The general normal equations can be solved for the v + 4 unknowns 
written across the top: SiS | methods of solution will be taken Me 
in sections 1l2b,; 17, and’) £0, but for the prescut we Shall only note 
that once the residuals vz, vp, ahd v, are found, the final (adjusted) 
values of the parameters are obtained by subtracting the residuals from 
the approximate values, thus: . 


A, Ror ava Wh OE iba evoke Came ce tene (62) 


It may not be superfluous to rémind the reader that the final 
peusbed values of a, b, and c are not dependent on the approximations © 
Ao, bo, and ¢,, provided these approximations are within reason, To be 
specific, if ay, bo, and cy are changed slightly, and a second solution 
mads, the residuals Vg, Vp, and ve will turn out to be slightly differsnt -- 
just enough in fact to bring a, b, and c into agrcement with the firs’ 
solution, Sometimes the approximations need not be close, yst the solu- 
tion is still valid; examples will be found in section 19, for instance 


on pages 119 and 133, and elsewhere, 


The solution of the general normal squations yiclds also numsrical 
values for the Lagranse multipliers Ars Aes wee 9 Ay, Which through 
Eq. 58 enable the residuals V; to be calculated. The observations Xy 
are then adjusted by subtracting the residuals, thus, 


ee ey SG. (cale'd = obs'’d. - residual) (63) 
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These adjusted quantities x,, along with the adjusted parameters 
found by Eqs. 62, will satisfy the v conditions expressed by Eqs. 505 


page 32, or their equivalent, Has. 53, page 35. 


Note by Eq, 58 that the residual V; is eM proportional 
to the weight w, of the corresponding observation, Does this seem 
reasonable? If any observation is relatively infallible, having 
w = 00, then its residual Vi : O: i, 6,, there is no correction. 

In curve cri tor example, it sometimes Pa OponS that all the x 


coordinates are'free of error; the corresponding residuals are thsn 


0, and the calculated values of x arc the same as the observed, 


s&4Q ~ 
[See. 10] 


i Ae = = EI 2 es 
10. Short expressions for S*. The normal squations are really 


normal, their cosfficients symmetrical and positive definite. 
Pe nnnnane SAVVIS DIU PLU PIPL 
By definition, ¢* = 2w,VF’ (49) 


Then by making use of Eq. 58, 
fe Sus ee 1 “Bes Viplly. 
pe = % Wy [A,FI* = wulakaly Agi st fies eee ae 


Len Pee AgR ESS iis Ayia le + =(haFE + AREY +) ,.2 
3 


We 
2 L heels - 
* Ropes A case — (AaFon + hel gn + vee * Lg 
en 


Another way of writing this is the following: 


Na Aue Ag eee My Vo Vb Ve 
Ee aa as Sa a 1 
Na Lia Lea Lgi eee Lapa @ Fy Fo 
Bil Seowarts bea\, Lwin ect Lae FZ Fe =F 
: 3 3 3 
As} Lis Les  lss me Lys y. Fy Fo 
ce : : (64) 
Vv 
ep Law) Leal dy Day Lyy Pa Fp Fg 
ve Fa Fe Fa . Fy 0 0 6) 
AD 2 3 Vv 
Yo} Fp Fp Fy Fp 0 0 0 
a oe Fo Fo So eae on 0 0 0 


> 2 3 
= Aiko + NeFo + AgFo + ra Aule = [A,F 3) by Eqs. 61, page 37. 


(hee g@= [A,Fe] (65) 


For special expressions of ¢* in curve fitting, see pages 109-111, 
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Since ¢% > 0, the quadratic form (64): is positive definite;* 
no matter what values be given to Ay, Ag, ..- » Avy Va» Vp» Ves the 
quadratic form (64) can not be negative. The symmetry of the general 
normal equations (section 9) has already been noted; hence they are 


really normal, 


11. Geomstrical conditions, or no parameters, (a) The normal 
PAPE DLO IV PLIVY PUT FUE von UU UU Se ae 


equations. If there are no adjustable parameters, then 


mh 20 h 
Ps; Fp, Fes Yas Vo» Ve, GO not sexist, and of the general solution 


(section 9) there is left only the coefficients in the upper left-hand 





Comune x < 
Na Le Ng Rats es = 1 
Te ia oS aos ie ane Liy Fo 
pares Ces nee Ley FS 
Toya RS il is ame (66) 
Lyy FO 


"Here the coefficients below the diagonal have been omitted, since in 


the abridged solution soon to be learned, those below the diagonal are 
not used. The coefficients are to be read "down to the diagonal, then 


to the right." The unknowns are the v Lagrange multipliers. 


a ae aa es es ase pk ae (cog Mae er a (ae? wear Ss saa ogn wi Sa Nes ga (SSIS ew mem a ee, em (ots ei ee See en eer eae ae) SP EE cen nee OS SR 


* Maxime B6cher, Higher Algebra (Macmillan'1907) p.°150. 


ih Oo. 
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This type of problem was solved by Gauss,* and is treated satis- 
factorily in many texts. It arises in geodesy, surveying, and in 
astronomy, and this accounts for the attentions of Gauss, Bessel, and 
Encks, who were mainly interested in the problems of adjustment aris- 
ing in astronomy, Unfortunately they did not give so much attention 
to problems involving parameters, especially when more than one co- 
ordinate is subject to error; and Kummell'ts paper (1879; see section se 
seems to have been overlooked by modern writers, 

It is essential to realize that fundamentally the principle of 
least squares applies is aaa a same whether there are or are not — 
adjustable parameters in the arenien This fact has been woefully over- 
looked. Critics of least squares have on more than one occasion in- 
Sisted that least squares was reasonable enough in the problems of 
adjustment arising in surveying and astronomy, but that it is illogical, 
equivocal, and unjustified in curve fitting. Statements such as this 
arise from confusion and misunderstanding, for there is no distinction 
betwecn the two problems. The student is urged to read the papers by 
Stewart and Uhler cited earlier (section 1), wherein these thinzs are 


carefully discussed, 


a  Deteetitieitintiantiendicetions 


* Ses reference to Gauss in section 9, p. 36. 




















SS 
“i . . 
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Example, The plane triangle* 
4 
aa The angles are measured with a transit. 
od a 
om ¥ Observations:: X,;/ Xe,: Xs 


‘\ 
; bs! Weights: Wa Wo4. Ws 


Pee 3 


Fie 4 Cale'd values: X,; Xe, X3 


(The weights might arise from the number of repetitions on cach anzle). 


This is a very simple problem, There is only the one condition 


Stee + he [corresponds to Bq. 50, p. 32] 
So hers we put 

eet. flat mak, ka tees LOO” 
(There is only the one condition, so no superscript on the F is needed), 
This F will be zero when (next page) we are able to insert the ad- 
justed values x,, X,, x3 into it. By inserting the observed values 
we have 

amen. +) Xe tReet 2 180° [See Ea. 5e,.p. Gol 
Fo is not zero unless X, + Xs + Xs happens to be exactly 180°, in which 
case no question of adjustment arises, The derivatives of F are 


Pate, a Fs 1 see Baw 51,Up i. S33 


> 


There is only one L (why?). It could be called L,, but no 


subscript is needed, so we shall use simply L. 


ene .ohat ase hals orl Sy os [Eq. 60, p.37] 
Wy We Ws Wa We Ws 


* A more complicated geometric problem, worked out numerically, 
appears in the next section. 


ae 44— 
[See. lla] 


Solution: = Fo/L 


The numerator F, is the amount by which the observed angles fail 


i 1 1 : 
to close. The denominator L is eer a enw ria hich happens to be equal 
P= 3 


to -— by Eq. 48, p. 27 (propagation of m.s. error). 
F 


After A is worked out numsrically, we find the three residuals by 


.Hq. 58, page 36: 


The adjusted angles are then Xi - Vii Me = Xe = Ves 2a = Sg ee 


val 
loa 
it 


Note that the sum of the adjusted angles is identically 180°, for 


Ky + Ky Fim Ky d Ko + XV ev 


v4 


1 i ts K,+ X, + Xz =1809 
ST hgh ieee ae eek “i 1 


Wi, We. . Ws 


180° exactly 


Note that the equations for V,, Veg, and Vz are valid no matter how 
large Fo is.. This is a case where there were no higher powers of the 
residuals to be neglected; cf. notes in section 9 and elsewhere, 

Note also that the residuals are inversely proportional to the 


weights of the observations; that is, 


Vip N ger a Seen Sea 



































-45- 
(See, lla] 

‘Thus in this problem the process of least squares simply takes 

the excess or deficiency F, (which will ordinarily be a small amount, 
perhaps a few minutes of arc) and distributes it among the three angles 
in inverse proportion to their weights. The student should reflect on 


this at length. If the action of least squares seams reasonable in 


. this simple problem, it must be so in more complicated ones,,even if 


we are not able to visualize its working so easily. The principle is 
always the same (the minimizing of % w res® or of x*); it is only the 
conditions to which the adjusted values are subject that differ from 


one problem to another. 


Exercise 1. Show that the condition 

x74 XS + xy = LEO? 
determines a plane distant 180//% 
from the origin, and cutting equal 
intercepts from the axes. The 
calculated point lies on the plane, 
and the observed point off it. If 
the weights are all equal, the 


distance between the observed and 


Fig. 5 


calculated points is to be minimized, 
in which case the line segment joining the observed and calculated 


points is perpendicular to the plane x, + Xg + Xs * Leo? 4 
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Exercise 2, All possible plane triangles are represented by points on 
this plane for which x,, x2, and xz are positive. Any method of adjust- 
ment would consist of picking off some point on this plane, correspond- 


ing to a given observed point X,, X,, X, off the plane. 


Exercise 3, If the weights of the observed angles are unequal, the 
distance between the observed and calculated points is not to be 
minimized, but rather the quantity 


wi(X, - x,)* + we(X_ - xX2)® + wyg(Xz - x3)”. 


Exercise 4, Solve the triangle problem (p. 43) without the Lagrange 
multiplicr. 
Hint: Take 9% = w V*% = w(X - x)* 
By the one and only condition on the adjusted values, we may take 
TROP L BO OH eI Ax, 
or Ve ="Fo =I g° = Ve 


where, as before, Fo = Xa + Xg + Xz - 180°, Then 


d2 = wiVi + weVP + wa(Fo - Vi - Va)? 
X, and x, are independent; so are V, and V,. Hence we may set .. 
dg2/dvV, and a¢2/aV, both equal to zero. The result is 


WiV, - W3(Fo - Va - Ve) 0 


WeVg — Wz ( i ) = 0 


It follows that w,V, = we.V,z,, and that 
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Viz (1/w.) Fof/{i/w, z 1/We i. 1/ws } 
2= (1/we) Fo/{ y } 


Vs= (1/ws) Fo i r } 


_. which is exactly the same as was obtained carlier with the Lagrange 




















multiplisr, page 44. 


Note all problems in least squares theoretically can be solved 
without the use of Lagrange multipliers. Occasionally it may even seem 
easier to dispense with them, but the truth is that most problems then 
become hopelessly involved. The elegance and uniformity that they 
lend to all problems seem to me sufficient to displace all other pos- 
sible methods in the design of a routine procedure. If Xummell in 1879 
had introduced Lagranze multipliers he would surely have accomplished 


the general solution that he was looking for. 


(b) The plane triangle continued, The weights of the adjusted 
angles and a function of them, Returning to the triangle problem of 
part (a), suppose we ask for the weight of angle -x, and of the sum 
Soe a, (* Xz, alter adjustment. Of course we know in advance that 
the weight of this-sum must be infinite, since we forced it to be a 
definite amount, 180°, but it will be Hiigresrinakd see if this 
result comes by the routine about to be described. The rules for 
finding the weights of functions of the adjusted observations are 


illustrated in what follows, and a morc complicated example will be 
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{[Sece, 11b] 
worked out in the next section. The theorstical proofs will be found 
in several books on least squares, for example . 

O. M. Leland Practical Least Squares (McGraw-Hill, 1921) 
T. W. Wright and J. F. Hayford Adjustment of Observations 


(Van Nostrand, 1884, 1906). 


Let G” = x, and G2 = x4) # xyt xyes mind Gare: the) fumes tore 
whose weights are wanted. As many more functions could be added as 
desired, but here we shall be content to sec just the weights of x, 
and of x, + X, + xz worked out. The procedure is as follows. We nee 
to form certain sums, and to this end one can make up the following 


table, numerical values ordinarily being inserted in place of the 


symbols in the body of the table. 





for 
numeri - 
cal 

check ~ 








pee re 

G>G id G:G 
: ivilj= ; a he L Ui n oi " 
Wy Wo i Wy 


{ ] means summation, as in scction 9; Gauss’ notation. 
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These sums are appended in the c* end Cc? columns, and the solution 


proceeds as shown under "How obtained", 





No, IN = 1 aly ca 
I L Bo (F,G;/w,J] = 1/wi [F,G?/w,] = L 
' ube Ene 24 
R 0 [e;G;/w,] = 1/w, [o5c3/w,l = 1 


How obtained 







Begs Tx { -1/w1,) 


S Eq. I x (-L/L) 


Bee Ue + 5 












In a numerical solution a Sum column would be used or a check on 


Eq. II, and the spaces filled in by the ellipses would be filled in with 
numbers, everything below Eq. 2 being produced according to the directions 
under "How obtained": for a numerical illustration see pages 64 and 65, 
Eq, I gives A= Eo/L as already found at the top of page 44, Look- 
ing next at the "1" column in Eq. II we see -FF/L or -AFo, which by 
Eq. 65 on page 40 is none other than hg, thus >); wV* is computed in a 
routine manner without first finding the “individual residuals Vay No yee var 
The variance coefficient of tse or the reciprocal of its weight, 
appears in oie obs eo ian of Eq. II*; and the variance dearricient of ee, or 
the reciprocal of its weight, appears in the c? column of Kq. jit Ps 


Before adjustment, the weight of G* was w,, the weight of the obser- 


vation X,: after adjustment, the reciprocal of its weight is 1/wy - 1/Lw,? . 


[Sec, 11lb]} 


Now of course 


a 
chats) emg 
Wy Lwf Wy 





which means that the.weight.of x, is greater than the weight of X,. 
That is, the weight after adjustment is greatcr than the weight before, 
which seems reasonable enough; the observations on the other two angles 
help to estimate x,, and to increase our confidence in its value. 
After the adjustment we feel that we know more about the triangle 
than before, 

In particular if all three angles have the same weight before 


adjustment, then if w, = w, = wz = 1, the weight of x, after adjustment 


is 1/{1/w, - 1/LwZ} = 1/{1-1/3} = 1.5. In this case the weight of ran 
is 50 percent greater than the weight of X,; i.e. the adjustment in- 
creases the weight by 50 percent; and the same is true for the other 
angles, 

If the weight of an angle has’ becn increased 50 percent, its 
S.£. has diminished 18 percent, since 


F nays _ [sec Eq. 16 
“pefors * Marter =(S-B-arter + S-Esperore)® Dp. 18] 


At this point the student will find Art. lea er whiteaker ond 
Robinson's Calculus of Observations to be enlightening. On page 254 
they show that in a quadrilatcral, all angles having been measured with 
equal weight, each adjusted anzle has 4/3 the weight of its observed 
value. 

Next consider the weight of x, + Xp + X,. From Eq. If in the form 


above we see that the reciprocal of its weight is zero; in other words, 


: ee j SCN 
<i (Sec, 11b] 


iT 









as simple example gives a glimpse of the method for the solu- 
tion of problems involving rigorous conditions. A guide for systematic 


computation, and a more complicated example, are given in the next 


section. 


Exercise. Take the values of V,, Vz, and Vg found earlier, 
namely A/w,, A/we, and A/wz, and show by direct substitution that 
Fo, the left-most entry in Eq. II of the tabulation shown above, is 


actually ¢* or w,V% + weVZ + waVi . 


LEO a 
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12. Geometrical conditions, continued. (a) Systematic computa - 
DAN 


VLISMM MM LISLE ASSIS PISIMS UPITIIV. 


tion; the weights of functions of the adjusted values. 


Observations: 1%,, Ae, .s5 ven (Here there will be no sub- 
scripts n+ 2°'n + 2 eee 
Weights: Wis We, seo, Wy since there are no y coordi- 
nates) 
Bondi tions: Wr" (a ee a ee) feng 
ie ee 2 5 i! X,) = Q (These are Eqs, 50 page 32 
. except that here there are 
Agee, Pax are hs x) = no parameters). 
4 
UR oerert ssa <. huree 


The lst stsp is to write down the conditions, i. ¢., to select 
the appropriate F functions; also the G functions whose weights are 
wanted.“ One then works out the values of Fo, which will usually turn 
out to bse small numbers, since the conditions will be nearly but not 
quite satisfied by the observations; sce for instance page 59, 

As here indicated, the solution will be illustrated with four 
conditions; i. e,, the number v in Eqs. 50 on p. 32 is taken as 4 
which will be the number of Lagrange multipliers (A). Expansion or 
contraction to more or fewer conditions is easy. (In the simple tri- 
angle problom of section 11, page 43, there was only one condition, 
and one A). 

We shall assume here that we want to find the weights of two 
functions of the adjusted values. Let these functions be designated as 
* By the use of tha reciprocal matrix da expleined tn eecuion | ame 


Sd 
one nsed not decide on all his G functions at the start; more can 
be added later without grseat inconvenience, 


— 
































LHS. 
: [See, l2a] 


Bi ’ 
S59. fore: hip bake cone nay ied Me ED SARA 


2 


GUX1, Xe, oes x) 


(In the triangle problem of section 11lb, G” was x5 and G” was 


X, + Xp + Xz; See. page 48). 


The 2d step is to work out the analytic expressions for the 


oF? jatar? agi 
Piste JHhLBO Gynmaee ijacae aetna 


a ae or a dx 











A * : Als 
derivatives such as F, = 


2 


o. dG 
Ba‘ Ox, 





5, 66C.; see, for instance, pages 60 and 61, 


The 3d step is to work out the numerical values of all these: 
eet eati vas: see, for Shatenes, page 61. In each case, the observed 
values Rehan pee are -useds in plade-of ayy Kay eg Rys since 
approximate values of the derivatives are usually close -cnough; atleast 
they will have to suffice till we can gét better ones, The following 
table is made up, numerical values being inserted in the spaces, 
Naturally, more or fcwer columns will be needed in various problems, 
and different computers will work differently even on the same problem, 
For example, one might wish to record w, in the 2d column and “put vw, 


in the 3d, Only general directions can be given in advance of a specific 


problem, 


Table 1 
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The 4th step is to form table 2, derived from table 1 by dividing 
the F and G derivatives by the corresponding Vw, as indicated in tne 


headings of table 2, 


Table 2,--The matrix for the formation of the normal equations> 


| a 1 
Fy Fy Fy Fy Gy 








2 
Gi | Sum 


The sums across tne bottom and at the right are used for checking the 
formation of the normal equations, With a machine having two multiplier 
registers, one for cumulating the quotients, and the other for reading 
individual quotients, one may cumulate one sst of sums without extra 
effort; see also page 101. 

oth step. The normal equations 67, together with the C+ and C2 
columns, are now made up by summing squares and cross-vroducts from 
Table 2. Thus, L,, of Eqs, 66 is the sum of the squares of the n 
entries in the P5 Vw column; L,, is the sum of the n cross-products 
in the FiVw, and Pi, columns (refer back to Eq. 60, D. 37), 
Instead of writing L,,, bie, stc., as was done in Eqs. 66, it is better 
now to fill in with the symbols that refer back directly to table 2. 
{ ] denotes summation, as before (Gauss' notation), 


a ee ee 


* Concerning the usé of the term matrix here; ‘see the footnote on p. 161. 
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Normal Eqs. 
| No. A Ne Ag ; Na = is C 4 oF Sum 
tent sie ee as shears 
| r ae a | E a ped Me ay ey 
i} — fe) eee 
We CLG ye b | i Wi L Wi 
| out 2s 2 4 he hy Bue 
| Wi | | Wy Wy Wi Wi 
ae Soe jee Oe Bem n 
3 Nite BiPilm aotp? Bi Sy B4S4 = (67) 
w Ww 7 Wi WwW ae 
| NX a4 ae 1 1 5 = 
eC > 
4 Qy a 4_4 ne at 42 8 6 
ae. : Sg a a a, ws - a ie f g 44 
s a Lop, Wy Beis Wy Es 
Pep. Q ) 
ce, F8y Peg ies wee 
5 g e On 0 pe ee 
Wi L Wy 


The Sum eam at the right checks the formation of the normal 
equations. Herein are entered (in pencil) the cumulation of the cross 
multiplications formed with the Sum column of table 2; these should 
| agree with the sums of the terms in the normal equations, the ABAH 
column excluded; see table 3 in part (b), and the check formed immedi - 
ately below. If no errors are found, the sums entered in pencil at 
the right of the normal equations are altered to include’ the "1" column, 
and the solution proceeds, being checked at the pivotal points (see 
the check marks in Eqs. II, III, and IV of the numerical solution in 


part b). The sums [GjGj/w,] and [cick /w;] must be checked otherwise, 























as by repetition. 
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The O in the bottom row of the normal equations is appended for. 
the computation of #*; the columns C” and C” for the computation of the 
weights of the functions G* and G?, | 

The solution of the equations is carried out by the routine 
process already seen in simplified form on page 49, and to be illustrated 
more fully on pages 64-65, and symbolically on page 107. When the 
numerical values of the Lagrange multipliers (A) have been worked out, 
the residuals Vi, ... , V, are calculated by Hq. 58 and then used to 


n 


find the "adjusted observations" x,, Zy,.++, mm as follows: 
iE apo 2 3 4 
Maw Vag Pavia y= Aa sam (ArFa + AeFa + AgPa + AsF1) 


il 2 3 4 
Kop" tegen ve = aa = Wo (Aske * Agha tiels © Adlag 


1 2 3 4 
Xn a x or Vy == Xy baa Wr (AnuFR as Neon be Ask + NaF y ) 
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(b) Numerical example--a surveying problem, A surveying party 
measures the sides and angles of the plane triangle P QR, with the 


followin: results: 


P 
On anzie P: 51° 06' 
08 
05 
06 
Average DOG Nees 
On angzle Q: 95° 05° 
Ce a 3 
Average GSS 04 tom 
Onvangiesk:. 33°.49' 
50 a} 


Bo 40's 





erae- ps Lfco.r. 10, 


WON &P Be05 bah /n 


The transit man, from previous experience, has reason to believe 
Piet.tha r.m.s. error of single measurements on one angle is about one 
minute of arc, or .00029 radians. He takes the r.m.s. error of the 
chainmen to be one foot in 10,000 fest, and in proportion to the square 
root of the distance chained, The weights of the observations on the 


angles and sides are then in ratios as follows: 


Wot Wg? Wp i Wy t Wy t Wy = 470,000: 29": 270.000,29% 


2/0.000 29% -: 10000/1724 : 10000/2205 : 10.00/1233 
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Looking back at Eq. 13 on page 12 we see that what has just been 
applied is the fact that wp is inversely proportional to the variances 
of f. The factor of proportionality is o*, the mean square error of 
observations of unit weight. Since weights are relative and not abso- 
lute, this factor of proportionality (o*) is arbitrary and can be chosen 


for convenience; accordingly we let 
o® = = (0,000 29)* = 4.23 x 10 © 


whereupon the weights take these simple values: 


Wp = 2, Wo = 1, We = by 
= ih) = f =68 = ae 
Wy 24 Gar TO "s Wq Lee eee S Wy S405 5 10 


It should be noted that the final adjusted values of the sides 
and angles, also their S.Es., are in no way depondent on the arbitrary 
choice made for o* ; if o* is doubled, all the weights are also doubled, 
and the S.Es. of all functions are left unaltered. Likewise x* is 
unaltered, 

lst step. The adjustment must be carricd out to enforce the fol- 
lowing three geometrical | 
Conditions Sine . s§intQ) sink 

p qa r 


P+@+R = 180°+ € 


€ being the spherical excess, which, owing to the small size of the 
triangle, will here be taken as zero, “If it were other than zero, FS 
(below) would be altered by the amount oe and the adjusted values of 


the sides and angles and their S.Es. would all be affected in an ob- 


vious manner. 











‘i 


sag. 
[See. 12b] 


For forcing the three conditions, let us set 


ete 8. oc, 4, r) = (sin P)/p - (sin Q)/q 


-_— 
= 
~ 
3 

-~ 
= 
= 
= 
- 
= 
= 
= 
—— 
i 


(sin P)/p - (sin R)/r 


P +.Q + -R°- 180° 


(The number v of Eqs. 50 is here equal to 3) 


The observed values of P, Q, R, p, q, r, when substituted into 
F*, F#, F® do not give zeros, but give the small numbers Fi, F2, F3, 
which by direct substitution are found to bse: 
FS = sin 51° 06'.25/1723.7 - sin 95° 04'.5/2205.4 


eel wen x LO. 


F2 = sin 51° 06',25/1723.7 - sin 33° 49'.5/1232.7 
=-0,5416 x 1077 
FS = 51° 06'.25 + 95° 04'.,5 + 33° 49'.5 - 180° 


= Ouro, 25 = 7 (27x 107° radians 


if ity had happened that the observations satisfied the condi- 
tions exactly, then Fg, FZ, and F% would have turned out to be zeros, 
and the adjusted valucs would have beén identical with those observed, 
As it is, the observations satisfy the conditions nearly but not 
exactly, i.e., Fo, Fs, and Fa are small but not zeros. 

F5 is the amount by which the sum of the angles exceeds 180°, 
In the simpler problem wherein the sidss were not measured (vide supra, 
section lla) it turned out that the least squares adjustment was simply 


an apportionment of this discrepancy among the three angles in inverse 


Mer on 
[See, 12b] 


proportion to their weights. Now, however, the sides are involved, 
wherefore the adjustment, though as reasonable as before, will not be 
so easy to arrive at. By looking ahead at page 67 we see that, in con- 
trast with the residuals on page 44, the adjustments on the angles will 


not now be all in the same direction. 


Now suppose that for some reason or other we should like to know 
the weights of 
Angle P 
The sum P + @ 4 Rk 
The area of the trianglc, which may be 
‘ L : 
sxpressed as 3 br sin @ 
Any number of others could be added (at incrsased labor) but three will 


suffice here, For those just named we take the three G functions 


Gt =P. G=P+Q+R, G3 = = vr sin Q 


ed step. The derivatives of the F functions are: 


FE = cos P/p FA" cog P/p PR ik 
Fa = -cos O/q Fn 26 | 3 Fa eal 
Fp = 0 Fe = -cos R/r Fp =l 
i ae P/p* usd th: fai yeh cae 
Fg = sin O/q? Fa = 0 Fa = 0 
Fi = 0 #2 = sin R/r® Fe = 0 


The derivatives of the G functions are: — 








Ae 
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ts 2 Me 
Gp F I Gp 5 ee 8) 
The other GA =1 GA = 4 or cos Q 
vu 4 G 
five are Gp =1 Ge = 0 
Zero The other G., = a2 sin.@ 
three are oa a0 
s 3 eek : 
ZEro Gy, 3 D sin @ 


3d step, The nearest numerical approximations that we can pro- 
duce for these derivatives are found by substituting the observed 


angles and sides into the expressions just worked out, and these will 


be more than close snough. 






4 


24.6°10~ 4,96+10° 
meis.2 " 4,38 * 


ae 34.5 " pom" 





Ath step Yw is now used as a divisor to form table 2 from 


table l, 


Aeon 
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Table 2,--The matrix for the formation of the normal equations. 





Bivw,10°  Fivw41l0® Fivwy GV, GW, G wy 107° 











P 0.257 6,707. 0.907, 0,707 0 
Q 0 i 0 1 -.094 1,946 
R -.674 1 0 1 0 1.326 
P -.528 0 0 0 1,238 0.182 
q 0 0 0 0 O- 468 
r 625 0 0 0 1.465 | 2,090 

eae 237 ~, 320 2.707 707 2.707 "2,609 EZ 
Re ee ee en, ee 


The powers of 10 in table 2 are chosen with regard to convenience, 
and to bring the number of decimals to uniformity from column to column, 
to facilitate the cumulation of squares and cross-products in forming 
the normal equations (the next step). At this stage one may also cut 
off superfluous figures, reserving, as a rule, not more than three or 
four in the largest number occurring in any one column. This often 
means that some other entries in the same column appear as zeros, but 


this is as it should be. 


oth step. The cumulations of squares and cross-products from the 
columns of table 2 provide the coefficients required for the normal 


equations 67 page 55. For instance* 





10° wan = 0.2578 2 . : i 
ore ~2D7%+ ,040"% + Of + . 528% + (468° + O = 0,565 


The subscript i will usually be omittcd for convenience hereafter, 
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as seen under A, in the normal equations. Also 





Siero 
10 ey Seo oro? (tl Ol=— g674 4.0 490 490.4 O = -0,492 


as seen under Az. The student should verify the whole set appearing 
in table 3. 


Table 3.--The cumulation of squares and cross products from 
table < for the formation of the normal equations, 















































se a - 0,565,) 10: ae. Groa5 5) tuo = 0,222 
; ‘ th be 
22 aps | 
10° ae 5 STOR Ni lem = -,492 
W WwW s 
3-43 
ae |- 2,500 
he Ww 
k=) Ros 3 F'G* Oe \=3 F°G* 
10 cml So. 1ee, 10 S| Veen, 10 We eo! 
" [pear wene2 a 2G3 
ake a |- 0.182, phe: = =-0,492, 10-« [Ee |- 0.262 
Vv 
Bad Ta3ae 6 {#303 
ey B00 po eres 00 Won ete Sie eee: 
’ | ’ 
_wW Ww W 
Lt 2n2 - G3G 
ao IF 0.500, a |- 2,500, "| : |. 3,687 
Ww 
Le 
-S- 878% 
W 
2 
foat: ir .994* 
i Ww 
[Fs 
— | = K 
|- shee 
Check”: 


ee cieumeAG | (eee + Lee ti kee - .657 = 2879 
feaoee 9h Ae + ee | 492 + 6 262)= .995 
eeu age) + 2,500 + .500 + 2,500 - ,094 =5.156 


—— ee ee we ee ea ee ee ee ee 
wee eee EO ee wee ew ew ewe ew ewww were — - 


No. 


Combined 


Factors 
.045/ .565 


.£22/ ,565 


.628/ 979 = 


7455/5565 
.027/ 5979 
.142/2.010 


.182/ 1565 
.071/ .979 
.474/2,010 


shee 14586 


.628/ .979 = 


2,.010/2,.010 


.657/ .565 


.663/) 5999 
.589/2.010 


wood 


i] 


6106 


ae et 


2004 
~Ver6 
0706 


pe 


LOT Re 





.06D 


(The powers of 10 written at the tops 


of the "1", C7, C®, and C*® columns 





LOK 
.345 


pa 6, 


Beau 
979 


ee 


a 


107*hs 


«sen 
-.492 
2.000 


solution of the normal equations, the 


i 


computation 


i) 


1 


~ 133xiom 


-,054 


a yo 


0 


.O8l 
*,027 


.052 
OL? 
142 


VOL 
=. VU0L 
-,OLO 
~,O&2 





- 308) 
073) 
OF L) VY 


\x1078 


sv) 


are understood to apply all the way 


down). 
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of TwV*, and the:weights of three functions 
ct ce C3 Sum 
Meeteexl0° 8 =. 222x107) - .657x10° 0.746Y 
nee, 182 ~,492 .262 0.941) 
500 2.500 -,094 5.209 / 
e500 | 2,500 1S. 687 6.573 / : 
How obtained 
Saba ~,136 .401 - .456 T(-.6106) 
071 -,628 663 485 / (2) + (5) 
BLO72 - 087 6258 eens I(-.3929) 
,046 -,403 .425 ns TI(+.6415} 
2474 2010 .589 5 227 (3) + (6) + (7) 
043 052 ~.155 176 I(-.2354) 
E002 -4017 -,018 Le es TI(+.0276) 
-,03% ~,142 -, 042 - , 369 III(-.0020) 
i) .508 2.427 3 APQ 6. 367V (4) + (8) + (9) + (10) 
Subsite curom Jel cca oom 
iy ts il aikanso} FUE 
IIt + 27010 
-,059 Tee oes 
-,005 vay Tt 0725) 
112 TII(-. 2358) 
324 CAy 41a eS pea 
-,087 I(-.3929) 
-.403 TI(+.6415) 
=2,010 | | TLC 
0.000 CAD Ey po Les ee Os 
764 11 =.1625) 
-.449 II(-.6772) 
Lew des TII(-: 2930) 
3,829 CA eee Ue Ue) he ee) 
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The sums at the bottom of page 63 aaa tally exactly with the 
numbers [F*S/w] = 0.878, [F*S/w] = 0.994, and [F°S/w] = 5.135 seen just 
above, but the agreement is within errors of rounding off, whereupon we 
conclude that the arithmetic in table 3 is correct, save for the three 
[cc/w] sums, which must be checked independently, as by repetition in’ 
reverse order; The cumlations shown in table 3 are then entered into oom 
Eqs. I, 2, 3, 4 of the tabular scheme for theaneesal equations: on the two 
preceding pages, The numbers entered in the "1" column of Eqs. I, 2, and 3 
come from the values of sy F2, and F5 on page 59 after multiplication by 
appropriate powers of 10 to produce decimals of the same aenominat ian as | 
the other parts of the normal equations, (The ree 107° applies to the 
whole of the "1" column). 
The sums at res eink of the normal equations are not the numbers 
879, .995, and 5.136 previously seen in the check under table 3 but are 
these numbers to which have been added the corresponding entries of the 
"1" column; the normal equations thus start off with a Sum column that 
provides checks at the pivotal points of the solution (note the check 
marks in Eqs. II, III, ana Iv), 
The Sau proceeds according to the directions under "How ob- 
tained", The Bene avatar of solution has been seen-in simple problems 
on pages LO, gateengians and will be seen again on page 107 and in 


section 20, 
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(c) Conclusions from the solution of the normal equations 
\1e From Eq. IV, 9? or LwV2= 0.042: 10 - 
It fcllows from Eq. 21 cn page 19 that 
G text)! = 0,042°107 4 (663) =) 1.4078 
Since this is only ahout one-third the prior o*® arbitrarily chosen on 
page 58, we conclude that so far there is no indication of blunders in 


the observations or recording. 


2°) Eqs. 13, 12, and 11 in the solution on pages 64 and 65 give 
Keer SORT AS = OL075, Ay = "0,071°10~ 
These used in Eq. 58 page 36 give 


Vp = B(AiFp + AeF5 + AgFp) =-0.0000075 radian = -0.03 min. 


Pp P 
(te oF + heFy + Neko Sr Pee cooecaanl (yg a) hey 
* a 2 “ 3 a >) 3 ’ 
Vp a NaF a Nok + Agk rs BOOOC ede 9 —~ 207 ' 
v. = (108/24..6) (AaFy HORSE A ASP OeesT 
p p p p 
= (108/19 P+ AeF,, + AsFq) =. --33 
Vq <4 (10 i p24 (Aux Fo ot Ne Ag q Leg ~.V0e 
Pewee S)OCr 2 Cr § eaLP } = | .o8 °" 
r Cin \iF r r 3*r ‘ 


for the six residuals. It is important to note that the derivatives 


required here are already worked out in table 1, page 61. 


“ 


3° By using these residuals with Eq. 63. page 38, we find that the 


Dad igated value of 


Angle P is 51° 06'.25'+ 0!.03 = 51° 06'.,28 


Prete GQ) is\95° 04".5. - .20 = 95° 04" .30 
Angle R is 33° 49'.5 - 07 = 33° 491.43 


ue eae 
[Sec .Lesy 
Side pis 1723.7---0.25 = 1725.45 Tt, 
Side .q is 2205.4 + 263°] 2205.75, 2" 
Side ris) 1232.7) - » 08) ="sesazocms 
Remark Perfect closure (3a condition on page 59) may be secured 
by lowering angle R by the trifling amount 0'.01; the. value 33° 49' .42 
so obtained, along with the other adjusted angles and sides just written, 
will satisfy also the lst and 2d conditions on page 59 to within 1 part 
in 4 million, which is about all we should ask for. Whenever, as hap- 
pened nere, one or more of the conditions fails owing to cumulated 
inexactness of See gree the computer is at liberty to manipulate 
the terminal figure of one or more of the residuals, raising or lowering 
it a unit or so to force the dun pices If not inconvenient, he will 
ordinarily (as was just done here) select the. quantities of least weight 
for any such manipulations, The amount involved will be small compared 


with the S. Es. of the final results, (CZ, also page 166). 


4° The weights and 8. Es. of the three G functions (page 60) are 


found as follows. 
6 +6 


From Eq. IV* the weight of the adjusted angle P is 1/0.324:10 “:10 = 


1/0.324. In other words, 0.524 is the variance coefficient of angle P. 


8 


Then with o®% = 4.2310. (page 58), it turns out that the S. EF. of the 


adjusted angle P is (4,23°10 °°0,324)” =1,2°107* radians = 0,40 min. So 


Angle Pois S1° 06" 0 2,0" 4 


mt eee ee ee ee 


From Eq. IV® the weight of the adjusted sum of P + Q +R is 1/0 or wm, as 


predicted. Hence the sum of the adjusted angles would be written 
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P + 4+ R = 180° absolutely 





From IV*® the weight of the area + pq sin R is 1/2.302 x 10**, Its S.E. 
ok 


is therefore (4,23 x 107° x 2.30 x 101”)* = 512 square feet; therefore 


the adjusted value of 


ihe area is LOS60ba + Sle sq. ft. 


= ee eee 





The area would better be written (105803 + 31) x 10 square feet, since 

not more than two figures .of the S.E, could be assumed known. In acres, 
The area = 24,2890 + 0.0072 seres 

(The area is found by using the adjusted values of p, q, and R and taking 

5 pa sin R. Of course one could as well use + qr sin, Por 4 pr sin Q 


for the area; one is as good as another). 


Exercise 1. Prove by Eq. 48.section 8 that after adjustment the weight 


of the area is a little more than double its weight before adjustment, 


Hint: By using Eq. 48 p. 27 we find that 


l/Werea = area” {1/p?w,, + 1/q?w, + (cot®R) /w,} 
ake x10 (103%) + 1.07 + 2.24} 
= O.60 x'10*" before adjustment 
a 2 

Ee Weran Tete x 10 (before) 


12 


We had Warea = Weight of Ge 0,43 x 10. (after) 


The stated result follows at once, 
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Exercise <. (From L, D, Weld's Theory of Errors and Least Squares, Mac- 


millan, 1916). -Take the line AB, on which are located points © and D. 


The whole line and its segments are measured with the same rule under 


Similar conditions, the results being 


X, = AC = 
Ag * AU = 77.90 ** now 
age Cn ee oe.) ee " 
Ka? CBS 8 ,obae eet 
Ae @ DE =te5759°°* " " 
he = AB =145.55 ¥" " n 


3 


a 


3 


a 


4 


" 


45.10 cm., mean of 2 observations 


Problem: Find the least squares values of the lengths, 


Fig. 


Také w, = a, We = 3, Wz = 2, W,. = 3, 


Conditions: F* = x, + Xx. + X, - X, = 0 
1 3 5 6 ) 


13 
F° = Xai = Xe * Xe = OC 





Show that the normal equations are as follows: 
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No, Ae Ne Ag = 1 Sum 
iis 2h 12 Pe hime 60+1072 105 

ie 16 6 108 " 142 

3 16 168 ” 202 

. 0 328 
_ Solution: Ay = 1.1145, Ne = ~.0952, Anu =u Doe 
- Residuals: V, = -0.0096 (by applying 


Eq. 58) 
‘ Pa teenie owe 
Vz = + .0680 
Voice e 6ORL? 


Mig weet COS 


Va = + ..0286 


Adjusted values: AC = 45.110 cm, 


AD = 77,992 
CD = 32.882 
CB = 98.412 
DB = 65.530 
AB =143,522 


— (AB actually turns out to be 143.521 cm., but the last decimal is 
‘raised one unit to satisfy the first condition. The other two condi- 


tions are satisfied perfectly by the adjusted segments). 


SFO. 
(Sec, 12c] 
Exercise 3, By Eq. IV in the solution of the normal equations of the 


preceding exercise, the minimized value of 2wV* is 0.0246. 


Exercise 4, Find the S. Es, of AB and AD, taking the S. E, o of 4& singe 


measurement to be 0,05 cm. 


Exercise 5. (a) Show that the estimate of o made from YwV* is o(ext) = 0.0! 
(ob) Show that with o = 0.05, x® = about 10, and P(x*) = 0.02, wherefore 

we might say that the discordance between the observed lengths of the seg- 
ments is somewhat larger than one might expect from previous experience, 
Note: Since the individual measurements were not recorded, there is no 
possibility of estimating o from the original observations; i.e. we have 


no o(int) to compare with the prior o and o(ext).: 


Exercise 6, The three inside edges of a parallelopiped are measured 
with calipers and a linear scale; and the volume is ‘measured in cubic 
units by filling it with mereury, which is afterward poured into a 


graduated cylinder, . The results of a set of observations are as follows: 





Mean -- n She Wate 
On edges parallel to the x-direction, X, (cm.) Ny 34.) 
On edges ae ae _ _y-direction, Xe daplidata Sp _ 
On edges Y n "z-direction, Ae aw enncy Glee hae 83 


On the volume, X, (ce.} Na Sa4 
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It seems safe to pool the S. Ds. on the linear measurements to 

get an estimate of their S. Es. or weights, and if n, + ng + ng is 
fairly large (20 or 30), this estimate may be fairly reliable. For 
as reliable an estimate on the S. EK. of the direct determinations of 
atime, ng itself would have to be as large as 20 or 30. Assuming that 
we have these numbers, or have gotten estimates by previous experience, 
we assign weights as follows: 

y= n,07/o? 

We = ngd*/oF% 

Wz = ngo?/o® 


w, = n4o®/of 


6, being the S. E. of a single linear measurement, and o, that of a 
Single volume measurement by mercury. o*, puna section 5, is an 
arbitrary factor of proportionality, the S. E. of observations of unit 
weight. If it is set equal to of, we should see the conreniees system 
of weights, 
Wir EW Walt) Wa * Nae Me ¢ Ne i? in,ok/og 

The weights Magi na been settled on, ws can proceed. The one 

and only condition on the adjusted values is that 


X4 = X4XeXz 


whence we put F XX, ~'X4Xo2Xz 
Suppose we need the S. E. of the volume after adjustment; we 


set 


~74- 
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(a) Show that the one and only normal equation is LA = Fo, whencs 
N= Fo/L, where 


L = X2 (1/x2w, + 1/%3w, + 1/Xfvg + 1/X4w.} 


(bd) In tabular form, the normal equation for finding A, %*, and the weight 


of the adjusted volume, is as follows; 














No, nS }: C 
Er oo (a 1/w, 
2 0 1/wa 
Ane Fok = ~ ie 


(c) The weight of the adjusted volume is (1/w,)(2 - iL) 


(a) (The S. E. of the adjusted volume)® = o£(1/w4)(1 - 1/L) 
Note that this is smaller than the S. E. of the volume before adjustment 


by the fractional amount 2 






(e) The minimized sum of the weighted squares of the residuals, ¢*, is FoA. 


(f) The estimate of o% by external consistency (section 6c) is 


of®(ext) = FoA/3 


(zg) What would you say if o#(ext) were much larger. than 0%, i.e. P(X) small? 
Sug sestions: Edges not parallel; lack of perpendicularity; measurements ~ 
not so good as initially supposed (i.e. 0, or 0,4 too small); just. hap- 


pened to be so, 


(h) Show that after adjustment the S, E. of the first edge is 


o/ (1 fyi eee 





BAe 


(a) Shorter method of computing the weights of a large number of 
functions*, The theory on which the weights of the three G functions 


were calculated in parts (b) and (c) rests on the fact that** 





‘GG n4G F°G F°G 
: pol ary Sey Gace i | Go Fr eserenraes ‘ 
-1/(wt. of G) & eae ake Alam } (69) 
where B', Bt', and B'" satisfy the equations 
ain 
gO eae ey tia) Bit) = a 
w 
Vee B+ Dee BY + ibe Bite ee) bahia, (70) 
é 3 
Doe iss Bis fh BIS a 


a 


In other words, the auxiliary constants B', B'', and B''' satisfy the 


normal equations 67 p. 55 with the "C" column 
F’G 
Ww . 
ad 
WwW 
F°G 
WwW 


replacing the "1" column... 





One may, if he chooses, solve for the Lagrange multipliers and 
any set of auxiliary constants B', B", B'' by first of all calculating 


the reciprocal matrix 


* To be omitted on first reading; the suggestion is that the reader 
return to this after a study extending through section 1’. 


** Gauss, Theoria Combinationis. (cited in section 6c) Art. 29, 
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Cad Cie Cig 
mene ‘ é 3 (see Exs. 2, 4, 
21 22 23 and 5 of section 17)-- 
C34 C32 Caz 


and then using it to calculate the Lagrange multipliers and the auxiliary 


multipliers as follows: 














Apa Eee. Sata, ieee 
Ne = noes EA Baas * Poces ( (71) 
Ag = Pees at PeCae 7 — 
F*G F7¢ F°G 
Bt = | Ww Jess i es Cia t - Ciz 
OPT aha eae re 


Bm = tt joa oF " |e + " Jess 


The Lagrange multipliers (X), after being calculated from Eqs. 71, 
are used in Eqs, 58 page 36 to compute the residuals V,, ..., Vn, just ae 
was done on page 67. The auxiliary constants Bt, B", and B'" from Eqs. 

72 are used in Ekg. 69 to find the weight of the function G, l1t will be 
noticed that the coefficients multiplying the c's in Eqs. 71 would already 
have been worked out by the first step outlined on page.5é and carried 


out numerically on page 59. The coefficients in Eqs. 72 come by cumulat- 





ing squares and cross products from table 2 of the fourth step, page 54, 


as was carried out numerically in table 3 on page 63. It is not difficult 
to extend tables 2 and 3 to carefor a new G function any time one is de- 


sired, 


The work then proceeds rapidly, the reciprocal matrix A-+ being used | 
over and over in Eqs. 72 for all the G functions, If one is working with _ 


a fairly good sized number of G functions, this scheme will undoubtedly - 














age 
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save considerable time over the direct computation illustrated. in 


section 1l2b. 


A distinct advantage of using the auxiliary multipliers is that 
the reciprocal matrix, once computed, is ready for use any time a new 
C column is produced, whereas with the direct solution in section 12b 
it is no little Pees to rye eR new C cone after a solution 
has once been carried through, 

The three G functions used on section 1l2b' will serve coe an 
illustration, ‘To calculate the reciprocal matrix A-1 we take the coef- 
ficients of the unknowns in the normal equations on pages 64 and 65, 
and put the unit matrix on the right of the equality sign, thus starting 


off with 


Po yee eee Ooyes eee. 10727] Fr, 0,0 


Pesoeem fee 190 |* 10%y, -)),498.» 10% = 0, 1.0 (73) 


Ree Ona. 492.6 10 oy +2 500. 1072/20; 0, 2 


The letters x, y, z simply designate the three unknowns that 
are to be solved for, Since there are three constant columns on the 
right, there are three different solutions. The simplest way to obtain 
them ie be to follow the regular routine foe solving normal equations 
illustrated in ssction 12b and written out in symbols in section 17, 
By whatever method carried out, the results written down in the order 


corresponding to Eqs. 73 give the reciprocal matrix 
Pe eet eva ve 1O® ~ 0.890.» 102 ” 
Pie eres 10%) 4. 296)* 108 .319 * 103 Th (74) 


asa)» 10% Vaio)» 10%) |, . 498 


The occasional failure 


=~ pee 


[See. 12a] 


of symmetry in the 3d decimal places 


comes from not carrying more figures; but what we have is good enough. 


Supposing that the Lagrange multipliers have not been worked out, we 


should next compute them from Eas. 71 as follows: 


~J.133°2,458 + 0,054:0.875 - 0,075-0,391 


-0,309 


A = = 
Ne SUH SEAS OTE A = w. 1.226 + " 0.319 = 6,095 (75) 
LOoks = Oe N00. o: « Bio BIghMa (el oreo mee 


These agree well enough with the values ~,308,:.073, and .071 already 


found in section 12b (conclusion 2°, p. 67). 


The chief aim at present is to compute the auxiliary constants 


B', B", B™ for each of the three G functions of section 1l2b, Going 


back to table 3 in section 12b for the coefficients needed for Eqs. 72 


we find that 


For G_ 

Bt = 10 {0,182-2.458 - 0.182-0.875 - 
Br = ote MOS CYE 9s oe haa cry 
BM = =i) 023904540" 5 Ol 8190+ 


These values used in Eq. 69 give 


el 


" 


1/wt. of G 
=O. 024 
That is, the weight of G* = 


section 120 page 68. 


0,500 - 0,182°0.0933 


1/0.324, 


0.500'0,.391} = 0,0933-10° 
" 0.919) (0.2245 10° > Ge] 
" 0,498 = 0,236 


0,182-0,224 - 0,500.0,236 
(77) 


in agreement with conclusion 4° in 
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Bo: a 
BY = 10°{ 0.222-2.458 + 0,492:0.875 - 2.500-0,391} = 0,00133.10° 
fe Of eh? 1d .a74%- Sov) peng + 9319} = 0,00028410% > (78) 
BM = os Cee aig +) 1 01498 = 


1.0015 


These used in Eq. 69 give 





2.500 - 0.222-0.00133 - 0.492-0.00028 - 2.500-1.0015 © 


{| 


1/wt. of G® 


-0.004 (79) 


Since weights can not be negative, we may suppose that this 
negative result is accidental from not carrying snough figures. The 
lowest possible result, if all figures had been carried, would be 0. 
Since we know this is actually what it ought to be, we shall not dwell 
further on it, but shall call the result 0, whereupon the weight of G* 


is infinity, as is already known. 


For G° As an exercise, the student should calculate B', B", and B” 


for G? in like manner, obtaining 


B' = -1,.808 + 10° 
B"'= 0,865 > 10° Hee, 
BY = 0,293 - 10° 

_ whereupon 

ae G® = 10°*{3,687 + 0,657-1.808 + 0.262-0.865 ~ 0,094-0,293} 


PAO LOT? (81) 





ee a at = 
re a ! o 


ES 


[sec, 124] 
in agreement with conclusion 4° in section 12e (top of page 69). Bs. 


Remark The number of auxiliary constants B', B', BM, etc. 


in Eq. 69 is equal to the number v of conditions, i.e. the number of F 





functions, This is also the number of Lagrange multipliers (A), the 
number of equations in Eqs. 7%, and the order of the reciprocal matrix, 
The number of G functions whose weights are wanted may be any whatever, 


smaller or larger than the number of F functions, 
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III. APPLICATION TO CURVE FITTING 


Loe Adjustable parameters enter | enter the | conditions, he problem of of 


curve fitting, The problem is as heretofore, to minimize X*. The 





solution is already contained in Eqs, 59-63, pp. 36-38, The L 

| factors now will be found simpler, consisting of fewer terms (compare 

| Eqs. 60 and 85); but parameters may enter these terms and appear other- 
| wise in the normal equations, There will be a function to be fitted-- 
a relation between x and y involving unknown parameters; however ex- 
pressed we transpose it all to one side of the squation and obtain the 


curve 











ye at, CO} aoO (82) 


This will be the calculated curve in Figs. 8 and 9, pp. 82 and 83. 


a, b, and ec are the least squares estimates of the true unknown 


parameters a, B®, and y. It is assumed that if there were no errors 
of observation, the observed coordinates X,> Y, Of Fig. 9 would 

| satisfy F(x, y; a, B, y) = O exactly. In other words, we proceed as 
if we are fitting the right curve, 

| For simplicity and definiteness, the development will be 


written out for two coordinates, x and y, at each point. The exten- 


sion to three coordinates is obvious, in which event Eq. 82, instead 





of being the equation of a curve in the x,y’ plane, is written as the 

















surface F(x, y, z; a, B, y) = 0 in the x, y, z space, See example 3 
of section 20 for an illustration in three dimensions, and 6xercise 24 


of section 19 for one in four dimensions, 
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observed 
x calculated 

0 true 

calec'd curve 


eyes: reset true curve 


Fig. 8. A Typical Situation in Curve Fitting 
It is assumed that the "true points" wherever they are lie on 
the "true curve" F(¥,%; a, B, y) = 0; a, 8, y being the true 
and unknown values of the parameters, The "calculated points” 
all lie on the “calculated curve” F(x, y; a, b, ¢) = 0; a, b, ¢ 
being the calculated (estimated) values of the parameters. 


The calculated points are estimates of the true points. 


(This figure and the next one appeared in an article by the 
author entitled "On the chi test and curve fitting", in the 


J. Amer, Stat. Assoc, 29, 372-382, 1934) 
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Obs'd Point 
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True Point’ 76 
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: 
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‘ ‘ 





Fig. 9. Relations between the "Truc" "Observed" and "Calculatcd" Points 


The x and y coordinates of a point are observed; these observations when 
plotted give the “observed point", The point that was measured is the 
"true point",which is unknown and lies on the unknown "true curve" 

F(R, 1; a, B, y)= 0. a, B, y beingthe true but unknown values of the 
parameters, The “calculated curve” is found by adjusting a series of 
observed points; its equation will have the same form as the true curve, 
but the parameters therein will be the "calculated parameters" a, b, c. 
Corresponding to each observed point there will be a "calculated point", 
whose coordinates are found by subtracting the "residuals" Vx and Vy 


from the observed coordinates X and Y, E, and E, denotes the "errors in 


y 
the observed point"; E,, Ey, U,, and Uy are unknown, but V, and Vy are 
calculated along with the parameters a, b, c by the method of least 
squares. As the figure happens to be drawn, each of the six quantities 
Ex, Hy, Ux, Uy, Vx, Vy is positive, Their signs are indicated by the 
directions of the arrows, 


=84- 
[See. 13] 
A point is observed to be X,, Yn, i.e. the x coordinate of some 
point ane Nh is measured, perhaps several times, and the mean of these 
measurements is X; with weight w,,. Likewise, the y coordinate of the 
same (true) point is measured, perhaps several Seite Aan the mean of 


them is Y; with weight Wyn. By Eq. 16, p. 15, wy, 3 Ww 


- yh = Var. Yy,: Var. X,. 

In order to make use of the solution already worked out, and to 
conform to the notation used in the preceding pages, Y, will be written 
Xn+h,s Wyn 2S w,, and Wyn 45 Wy+p, there being n observed points altogether, 

The conditions (50) will be n in number; i.e. the number v used in 
Art. 9 is now equal to n, the number of observed points, The reason is 
that for each observed point, there is a calculated point, and the calcu- 
lated point is forced to lie on the calculated curve; i.e. the residuals 
must be just the distances required to put the calculated point on the 
calculated curve; see Figs. 8 and 9, also Fig. 13,p,168. If Xs Sana 
(i.e, xh: Vy) is the calculated point corresponding to the observed 
point X,, Xp+p, (i.e. X,, Y,), then x, is the least squares estimate of 
En and x,+p (or y;,) is the least squares estimate of Mh: 

Now since the calculated point x,» Xn+th must lie on the calculated 
curve (82), then 

, Ext sen ee yee 

must vanish at every one of the n calculated points, Thus, Por every 
point there is one cont eon imposed on the 2n adjusted coordinates 
X1, Xn+135 Xe, Xpn+n3 «+635 Xn, Xey; Altogether, then, there are as many 
conditions as there are points, For n points there are n conditions, 


h 
So we set the F of Eqs. 50 identical with F(x,, Xn fh Ge MG Lee 


Ce enn 





























se Y 
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h ; 
Foie F(x; Xn+hi a, d, c) or F(x), Vu rey dy c) (83) 


Immediately we see that* since neither Xo nor Ye is involved at 
the point x}, Yn; 


Fe 


=O unless g=h or nth (84) 


It follows* from Eq. 60 p. 37 that L,, = O unless g = h, whereupon the 


& 
L terms off the diagonal in Eqs. 61 disappear (see Eqs. 86 ahead). 


For those on the diagonal Eq. 60 gives 


~  _ fh Ph , Fath Fath 
Luh = 
Wy Wh+h 


which is needlessly cumbrous in appearance. In the first place, since 
L terms will appear only on the diagonal, the double suffix is not re- 
quired, so hereafter we shall write simply L,; in fact we shall soon 


form the habit of omitting the suffix altogether, writing 


r FF 
ips Re tik a 


Wy Ry 





(85) 


it being understood that all quantities involved may vary from point 
mo point. The first term drops out if X is free of error, the second 
if Y is free of error; see Remark 2 in Ex, 4 of section 19, p. 128. 


By Eq. 48 p. 27 we see that L at the point h is actually the 


weight of F(X,, Yy; a, db, ¢) or of the quantity called S in exércises 


3, 4, and 5 at the end of section 15. On this account, W will 


* Ihave seen two problems in curve fitting wherein this was not true; 
the coordinates of one point depended by addition or multiplication on 
the coordinates of an adjacent point. One of these is worked out in 
the Phil. Mag. (London) 17, 804-829, 1934, 


“Rae 
[See, 13] 

frequently be written in place of'1/L as we go along, though it is 
better to perform the actual numerical calculations with L rather than 
with its reciprocal (cf, section 16 and some remarks in sxercise 4 of 
section 19). 

The general normel equations (p. 37) for curve fitting now take 
the form shown below in Eqs. 86. They ean quickly be reduced to a small- 


er set, Eqs. 88, in number equal to the number of adjustable parameters. 





Mex Ne Ng Pave Anteg ate Vp. ae aN he 
Ly 0 0 ve 0 Fa Fy Fo Fe 
0 f. 0 at 0 Fe ee Fr? F? 
0 0 hi 0 Fe ue FS Fe 
: : : , : (86) 
0 0 0 * ie Fe Fy Fo Fo 
EF Fe Hyes Lee i. . 0 0 0 
Fy Fy Fy 5 FS 0 0 0 0 
re, EB Fe ss ute 0 0 0 0 


Ary Ags soe y Ay Ore easily sliminated; solve for them in the 


upper n equations, getting 


Bf \ 
A = eatee - Fava - FLV 5 - Rev) 





ek 2 2 20 
EME Boum Fave aoP tae PoVaq) | (APA for 
| each .point) . (87) 
Fein | 
Ae = lane - Fav, - Fovp - aah 
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and then substitute these values of Ay, Ag, «.. , Aq into the lower 


three equations of (86), The result is 


























Vo Vp Vo = x 
oe ee 2 PaFo 
Ae ae ee | en, 
pee pe ee ani} (38) 
gal T; pst an 
ess) (eet) at) (ier 
ints fae nie 





These equations contain only the residuals v,, Vb, Ve as -un- 
knowns. The arrangement of the coefficients is symmetrical and their 
quadratic form positive definite, like the general normal equations 
whence they came (section 9), Once the residuals va, Vp, and Vo are 
obtained, the adjusted values of the parameters are found immediately 


by subtraction from the approximate values, i. e. 
b = bo - Vp (Ras. 62, p.38] 


As noted in section 9, the final values of a, b, and ec will be the 
same, regardless of the approximations a), bo, and ¢,. In some 
problems, dewereu, these approximations must not be too rough; in 
others, it makes no difference what they arc, except that more fig- 
ures are en tad in the normal equations; see exercises 4, 5, and 10 
in section 19. In regard to the matter of arriving at the approxima- 
tions 4), ei Co, see a footnote tt section 39 on the "method of se- 


lected points", also example 2 in section 20. 


[Sec. 133 


| 
93. | 
=| 

| 

Thus the calculated curve (Fig. 9) is fixed; it is given by ) 

} 
Eq. 82 with the values of a, b, and c just written. 

Several details remain: to adjust the observations (sections 14 
and 15); to work out a systematic procedure for setting up the normal 
equations and solving them (sections 16 and 17); esis are the 
weights, or the variance and product variance coefficients of a, b, ¢ 


(the reciprocal matrix, sections 17 and 18); a short-hand calculation ; | 


of the minimized value of #* (section 17). 








Remark It is well known that a rapid method of computing the 
mean x and the S. D. s of a set of n observations Xi, Xg, eee 5 Kny 
is to select first of all an arbitrary datum, perhaps a rounded off 
guess at x, then to write down the departures from this datum, and | 
their squares, taking finally their averages, and correcting the 
arbitrary datum to find x, and at the same time s. This is exactly 


equivalent to fitting the line 
ee (see section 3) | 


by least squares, the arbitrary datum a, being considered an approxima- | 


tion for a, The work might be laid out something like this. : I 






































i) -BqQuarenos, 
Observations Departure from a5 the departure 
(notation of section 3) from ao | 
Xy Xi - Ao (oes &o)* 
Xg X2g ~ Ao (xX, - 0)” 
Xz Xz - Go (x3 - Ao)* 
=A _Xy - Bo (x. ml Belk 





Av. = A (say) Av. = B (say) 
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x and s are calculated by taking 


X¥=a gta 


The student should peste that these equations are true to the defini- 
tions of x and s, no matter what value be taken for ap. The process 
is easily modified to take care of unequal numbers of observations 
Pes qual weights), 

Numerical examples of this procedure abound. R. A. Fisher's 
Statistical Methods for Research Workers contains many applications, 
and it is common in most all papers dealing with the analysis of 
variance, as the student is doubtless aware, An example is worked 
out in Whittaker and Robinson's Calculus of Observations (Blackie and 
Son, 1929) Art. 97; one also in Deming and Birge's Statistical Theory 
of Errors, The reason for mentioning it here is merely to emphasize 
that fundamentally the procedure amounts to a least squares solution 
wherein a, is an approximate value of a (actually a, need not be any- 
Where near the final value of a, but the computation is casier if is). 

After sections 16 and 17 have been studied, it will be interest- 
ing to return to the following tabular scheme, which is equivalent to 
the procedure just outlined for the computation of the mean and S, D, 
of a set of n observations. We form the sums (x - ag) and X(x - ag)” 
calling them nA and nB respectively, as above, and enter them in 


Eas. I and 2 below, performing on them the steps indicated. 


K9Qs ‘ | a) 
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isis Wk A uk. 
it rat —nA it 
& nB 0 
ONL SS a 
. | -ni? ... (Multiply I thru by 3) 
Lt mB - nt? .., (Add 2 and 3) 


Solving Eq. I for V_, we get v., = -A, whence the least squares 


a 


value of a is 


@ * Got Ves Got A 


i} 


Get p(X i Gahie we 

| Looking at the left-most entry in II.we see nB - nA*, which is 
none other than ns®, The tabulation seen here is a simple extension 
of the one given in section 4b, and a rather trivial application of the 
routine process to be learned in section 17 for the solution of normal 
equations and the computation of the sum of squared residuals in.curve 


fitting. 








i 
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14, Adjusting the observations, or finding the calculated points, 
A Ue ke 


PILL LESS PIILSELLILIUSPLLIVSLPIUIVIUIIPAA SLY UU. SUI 


Going back to Eqs. 58, p. 36 we see that the residuals will be 


ave eee ey (x residual at point h) 
wees: (89) 
Vy = w, Na ce (y residual at point h) 


The adjusted (calculated) coordinates can now be found; they are 


i 
x, = X, - Vx 
at point h({see bottom p. 38) (90) 


It 


These are the coordinates of the calculated point corresponding 
to the observed point X,, Pa Finding the calculated points is the 
process of adjusting the observations; when Xp and yy, have been calcu- 
lated, the observations X,, Y, are said to be adjusted. The calculated 
point Ayo Th is the least squares estimate of the position of the un- 
known true point or The Obviously, it will depend not only upon Xy> 
Y, and their weights, but also more or less upon all the other points 
and their weights. 

Just how is this dependence tied up with the other points? 
Through the normal Eqs. 86, or their equivalent, Eqs. 87 and 88: 

Eqs. 88 supply the parameter-residuals Var Vp? and va which are to 
Pee uisedin hos. O mM To tind Ay Ne iss 0 A These in turn are used 
in Eqs. 89 to compute the x and y residuals at each point, by which 


the observations are adjusted as indicated in Hq. 90. 
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We now have a method of adjusting the observations and of estimat- — 


ing the parameters a, b, c, when both the x and y coordinates are sub- 
ject to error, but it must be remembered that the solution depends on 
certain simplifying assumptions; see footnote on pp. 34-35, also the 
exercises of section ieee | 

A valid and familiar method of adjusting the observations, avail- 
able only when all the x coordinates or all the y coordinates are free 
of error, is to substitute -the coordinate free of error into the formula 
F(x), Vnv 8, D5 c) = 0, and solve for the other coordinate, Thus, in 
the parabola y = a + bx + cx®, if x is free of error it is easy to 
calculate y for a given x, once a, b, and c are determined, But if 
only y is subject:to error, only Vy and y;, need to be calculated from’ 
Eqs. 89 and 90. Likewise, if only x is subject to error, only Vy and 
xX; need to be calculated. When the function F of Eq. 82 is somewhat 
involved, even though only one coordinate is subject to error it will 
sometimes be easier to adjust the observations, i.e. to compute the 
"calculated points", by means of the Lagrange multipliers, through 
Eqs. 87, 89, and 90, than to substitute direttly into the formula 
F(X,, Ynys a, b, ¢) = O and solve it for the desired coordinate in 
terms of the one that is given. When both coordinates are subject to 
error, one must apply Eqs. 87, 89, and 90,-if he would adjust the 
observations, For a numerical illustration, see pages 165, heey and 
168: it is suggested there that the student work out the numerical 


values of the remaining nine calculated points, 
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It might te well to look for a moment at the parabola y = 
a+ bx + cx® again, and consider the difficulty of solving for x in 
terms of y, as one would need to do by the older method if only x were 
subject to error. Of course it can be done, but it would be easier to 
compute V, at each point by means of Eqs. 87, Bo. and 90. 

In the past, a great deal of emphasis has been placed on the 
parameters, Books and courses on aaiiatniens of observations have 
usually dealt with estimating the parameters a, b, c, hardly mention- 
ing the adjustment of observations. Least squares is primarily a 
method of adjusting the observations by minimizing y*. The parameters 
enter only as unknowns in the conditions that are forced upon the ad- 
justed observations. As a matter of fact, least squares is the only 
method of curve fitting that professes to adjust the observations, 

Now of course, the estimates of the parameters and their S. Es, 
are often the prime purpose of an investigation, But however important 
they seem to be, it is. well to keep in mind just how and why they 
enter the adjustment by least squares, in order to understand the 
results, 

This brings up another point of. interest. The estimate of o 
by external consistency; called o(ext): heretofore (section 6c) depends 
upon the calculation of g* or X(w VX + Wy FH). Neither the fit of the 
curve, nor the estimated S. Es.. of the parameters, both depending on 
g*® , can be calculated unless both the x and y coordinates are given 


their correct relative weights, 


~9ax 
(See, 14] 


It is a fact that no matter how many coordinates at any point 


are subject to error, the least squares value of 


PA eS lois é 2 
Ke ome EWN oe at (91) 


for the fitted curve has the probability distribution 
k-2 1 


Ay 2 
ply A) dyen= ot hee ye 2hke 2X ay ® (92) 





with k = number of points - number of adjustable parameters. 

The proof of the y* distribution in the gerieral case is contained in 
the paper cited in the legend of Fig. 8. A necessary lemma thereto is 
given in the London’ Phil, Mag. 19, 369-402, 1935. The effect of in= 
cluding both x and y residuals in x* is merely to increase the value 
of X* over what it would be if only onecoordinate were subject to 
crror, the form of the curve being unaffected, 

It is to be noted that there is no such thing as a distribution 
of X* unless the fitting is done by least squares; in other words, 
only the minimized y* has a distribution, (The assumption of normally 
distributed observations is basic to all these statements). 

The adjustment of the observations, or what amounts to the same 
thing, the calculation of V, and Vy at all points, is important; to 
know the sum of wxV,* + wyV,* is often not enough and it is then highly 
desirable to make a study of the individual deviations. See the dis- 


cussion in Example 2 of. section 20, 
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15, Some geometry concerning the adjustment of observations, 
PLE UV SSP PIP IP PAI WISI ULVR VULVA SUPIPUIUVAPUURALLVUVPUUTUPIULTU 
| Now let us consider some of the details connected with the calculated 


i points, or the adjusted 


me cis tangent pare observations, The ac- 
ovat point hi} mi 

aS ‘i f 

| ie , 

The Dyed Or, analysis are helpful to 


R 
| 

eo tS slope ay /ax / ; companying diagram and 
calculated \>, 
{ 
! 
| 





| pa y "an understanding of just 
| what is accomplished by 
: (or adjusted) \ the application of least 
| ‘squares. 
Let Q be the line 
Fig. 10. A portion of Fig. \\ segment joining the ob- 
| 9 redrawn for further con- Np Th CG are eet 





sideration. 
points. By Eqs. 89 we 


can find the slope of 


this line segment; it is 


! aF 
% GE 
Mm slope | the y residual Vy x By Wx OF | Na & eae 
Pen the x residual Vy Wy Fy wy oF Wy dy 
dx 
That is, 
aay 1 
The slope of Q = -(w,/Wy) —————————_______ (94) 


the slope of the curve at h 























eee me eee ee ee ee ee ee ee ee ee ee es we ee ee ce eee ee eee ae ee ee ee eee ae we we ee ee ee et ee eee ee ee ee ee se ee 


* The last step here involves the very important relation learned in 


elementary calculus, that if F(x,y) =c, then dy/dx =- a a =- F,/Fy. 
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Hence if at point h, w, = w. 


x y the two slopes are negative reciprocals 


of one another and the line segment Q is perpendicular to the curve 
(but see exercises at the end of this section). 

If at any point, Wy : Wy =o or is practically very large, which 
is to say that X is relatively infallible, then the line segment Q is 
vertical and the adjustment is all in the y coordinate, 

If at any point, Wy ? Wx 
is to say that Y is relatively infallible, then the line segment Q is 


=o or is practically very large, which 


horizontal and the adjustment is all in the x coordinate. 
If at any point, Wy ; Wy is finite, i.e, both X and Y are subject 
to error, the line segment Q will be neither horizontal nor vertical, 


but inclined, and there is adjustment in both the x and y coordinates, 


Exercise 1, With x and y measured in certain units, the weights of X 
and ¥ at a certain point are equal, and the line segment Q is therefore 
perpendicular re the curve at that point. Then the units in which vile 
measured are changed, as from feet to inches, Prove that the line 
segment Q is no longer perpendicular to the fitted curve. (Hint: the 
ratio Wy + Wy was unity before the change in scale, but it is not so 
afterward, If all the y coordinates are multiplied by C, then the 
weights of all y observations are decreased by the factor 1/c 2, and the 
new value of Wty 3 Wx is i/ce times the old; hence is no longer unity, 


C being supposed not unity), 
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Exercise 2, When there are three coordinates, the surface 

F(x, ya ay bi, cy i= 

is to be fitted to the n observed points, L trey woneeine three terms-- 
the two already written in Eq. 85 plus F,F,/wz. Show that if x, y, z 
are observed with equal Wolent at Ae point, the linc segment @ joining 


the observed and calculated points is normal to the fitted surface.* 





In such a problem, the calculated points lie on the calculated (fitted) 
surface, See the last part of section 20 for an example in three 


dimensions, and Ex, 24 of section 19 for one in four dimensions. 


Exercise 3. Let S be the value that F, would have at point h if ao, . 
Bo, Co were taken equal respectively to the final values a, b, ¢; i.e., 


in symbols, let 


S = Fy - Fava - Fuvy - Fove 
= FV, + FyVy . Gece Eg! 53, Pw oo) 
Sesto eC Dyc) 


h? 





Then O(w V,* + WyVy ) can be written Ews?, and this would be the value 
of g? calculated under the assumption that a = ag, b = bg, © = Co. 
Wis here written for iWae in Eq. 85. Hint: From Bas... 87, A\= WS, 


whenee Eqs, 89 give 


Nes hay) WS By 


=< 
u 


. (1/wy) We ee 


emer ee ee ee ee ee i ee ee ee eee ee ee ee ee ee ee ee ee ee ie eee eee 


i * This result was first stated in the Phil. Mag. (London) Ay) 246 
. 156, 1951. 


i 
Jk 
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2 


2 
for the x and y residuals at point h. Substitution into w,V, + WyVy 
gives the required result in terms of S (due to Kummell, 1879). This 


result is useful in Exercise 3 of section L7. 


Exercise 4, Prove that W in the preceding exercise is actually the 
weight of S, i. e. of F(X,, Yy; 8, b, ¢). (Hint: Apply Bq. 48,eme 27 
Hence the new expression © WS* for ¢? can be regarded as a sum of 


the weighted squares of residuals, S now being defined as a new kind 


of residual, 


Exercise 5, If the x coordinate is free of error, WS* is equal to 


WyVy, and if y is free of error, WS* is equal to w,V,2. 


Exercise 6. Prove that for any observed point in the neighborhood of 
which the slope of the fitted curve is positive, the residuals V, and Vy 
will have opposite signs; but if the slope is negative, then V, and Vy 
will have the same sign. In other words, when the fitted curve lies 
below the observed point, then the calculated point lies nator and to 
the right of the observed point if the slope of the fitted curve is 
positive, but below and to the left if the slope is negative; and when 
the fitted curve lies above the observed point, then the calculated 


point lies above and to the left if the slope is positive, above and 


to the right if the slope is negative. 
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16, Systematic: procedure for setting up the normal equations 
PPLE LSPLOSLIGIPLS PL IIIS PIP SSS PLIGG PU YY WU LILY PUTTAR 


for the parameters. Given the formula to be fitted, one can transpose 


it all to one side of the equation and have 
Fix yece. D.. c}' = 0 : (82) 


The rina step* te to work out somehow satisfactory approxima- 
tions a, bo, Co for the parameters (cf. re canaries in section 9; 
also te AES cake Fg at every Rane In some problems, depending on 
the formula and the hai hon it is permissible** to take Ao = b, = 
Co = O when calculating F, (but not L), in which event the residuals 
Va, Vp, Vo turn out to be the adjusted values -a, -b, -c themselves, 
But this is not usually advisable even when permissible. As a matter 
of saving time a good rule is to commence the adjustment with as good 
approximations 4,5, bo, Co as can:be found with a reasonable amount, of 
trouble, and thus to cut down the number of figures required in the 
formation and solution of the normal equations. | 

The second step requires some differential calculus; it consists 


merely in writing down the derivatives of F, namely 


F., and F 


Monat Sot hes : 


a? 


looking toward the calculation of | 


= FLF,/w, + FYFy/wy (refer eee Eq. 85 


ee a a a a a a a a a a a a a a a a a a a a a a a a a a a wr rrr wren rere 


Do) tosis ee ere to compare these steps and tables with those 


of section 12a, where there were no parameters, 


*K Seo Exs. 5, 10, 17, and 21 in section 19. 
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and the summation required at the n points. L may vary from point to 
point, and some or all of the derivatives F,, ae and F, almost surely 
Will.» So will F,. | 
The third step is to work out the numerical values of F,, Fy, Fe; 


and L, at every point. The following tabulation is suggested, 








Of course auxiliary columns may be required, depending on the 
problem and the whims of the computer. Or, perhaps some columns listed 
will not be needed, e, g. if w,, were oD all the way down (x free from 
error) then F, and w, would be omitted, since y alone A contribute 
to L, which would be merely FF y/Wy. Likewise, if y were free from 
error all the way down, then the Py and Wy ce Lanise would not be needed, 
for then L would be simply FF ,/w ; 


The fourth step is to divide each entry under Pas. hare and F, 


@? 
by the corresponding vL. The sums at the right or bottom (one but not 
both) of table 2 can be formed by cumulating these quotients in the 


horizontal or vertical, the individual quotients being entered in the 


table, (This cumulation requires a machine with a double multiplying 
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dial, one to be locked for cumulating quotients, while the other clears 


when desired. See a remark following table 2 in section 12a, page 54.) 


Table 2.--The matrix for the formation of the normal equations * 


(4th step). : 











| h oe — — — Sum 
| WL, VL Vi 

| : - : : ; : 

| ‘ ; . : 3 2 
i . 

| | 
n = . of i ra 
Sum - - = ae ad 


The sums at the right and along the bottom are used for check- 
ing the formations of the normal equations exactly as was done with 
table 2 in section 12a. First of all, the sum across the bottom should 


| equal the sum down the right-hand side, as indicated by the check mark, 





In running down the columns, cumulatiny squares and cross-products 

(the 5th step, next page), the final total in the multiplier register 
will equal the sum at the bottom of the multiplier column provided no 
changes in sign occur in the multiplicand column, In a machine with 


a double multiplier register, one part of which can be locked for cumu- 








lation while the other clears, individual multipliers can be checked at 
will in one dial, while the sum of the multipliers cumulates in the 


other one for checking at the bottom. 
Ae 


— on -— = on a lr lll llr lh lr llr lr ll rll rl rl Ol rl Ol Ol rll rOlrOOlcrOrOl Ol rOellll rll ell el rel el 





i * Concerning the use of the term matrix here, see the footnote on p, 161. 
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It is difficult to imagine circumstances wherein a maximum of 
three or four significant figures in any column will not suffice. 
This means that if there is great variation in the vertical in any 
column, some entries in table 2 may have only two or one, or not even 
any figures; see,for instance, pages 154 and aes The denominations of 
the different columns should be made Oe cara by writing powers of 10 
at the top of each row, to apply to the whole column (see the solved 
examples at the end; also the one in section 12b). No attention need 
be given to the powers of 10 until the end, when the solution of the 


normal equations is decoded (unscrambled,I have called it in my lectures), 


The fifth step is to form the normal equations (see the next sec- 
tion) from table 2 by the familiar process of adding squares and cross- 
products of columns. Thus, no Bit 9.5 wea Ate eet tie weighting, 
and no matter what be the form of the fitted curve, the whole procedure 
is uniform, and we are brought to a uniform and familiar process for 
the formation of the nOrmal equations. 

As already suggested, the student should compare this matrix with 
the previous table 2 of section 12a, which arose in the consideration 
of conditions not containing parameters. The headings in the tables 
are different there, of course; but the process of forming the normal 
equations from table 2 is the same here as it was there-- in each case 
a routine procedure. Also, the routine of solution is the same (com- 
pare sections l2b and 17). The exercises in section 19 will provide 
practice in the necessary steps for setting up the normal equations 


for several types of functions, 
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_ important note. By the procedure here explained for the forma- 
tion of table 2, whence the normal squations are to be set up, the 
solution (i. e. the values of vg, Vp, Ve,%", etc.) is unequivocal. 

That is, it does not matter in what form the equation tobe fitted is 
written, dig one had, forrexample, y = aeo™, he could put the same | 
equation in the form ln y = ln a + bx using in the former case 

F = aeX - y and in the latter case, f = ln a+ bx - in y. These 
things will be made clear in the exercises of section 19. The point 
that I am making is that when the normal equations are correctly made 
up, they will yield the same results from any form of the fitted 
equation, re within higher powers of the residuals. As another example, 
consider the straight line in the two forms y = a + bx, and x = -a/o+y/b; 
again the results would be the Lene to within higher powers of the 
residuals. Summed up, the results--the final calculated parameters, 

the equation is written,* “Many vext Hoole have been indefinite and 
ambiguous in meee of this nature, though it is a pleasure to 

record that Leland in his Practical Least Squares (McGraw-Hill, 1921) 
states clearly that the weight of In y is hae times the weight of y 


(see exercises 17, 18, and 21 in section 19). ° 


ee me ee nee me ae ee ee ee we ees ee ee ee ee ee ee ae ae a aa ee ee ee re eee ee een ee 


* Very large residuals, i. e., very rough data, will invalidate this 
statement; see some other remarks in section 9, also in Exs, 18 and 
ee of section 19, 
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: ‘ . 4 E ey j cj ocal 
eye Systematic solution of the normal equations. The recipr c 


matrix; Systematic computation of ¢*, As has already been noted, the 
NU ISII URLWIRE LLP SUV PUT UE 
sums of squarés and cross products occurring in the normal equations 88 


(p. 87) are formed directly from table 2 of the preceding section, Thus 


Pala 








F ; 
is the sum of squares under the column headed ot the summation 
PaFp] , : 4 
ral is the cumulation of cross products under the columns headed 


Fo Fp 
zr and 7 rete, 

The following systematic solution of the normal equations is the 
same one that was used in section 12b, save for slight departures toward 
the end to accomplish different purvoses; also it is the same scheme 
that was illustrated in the simple problems of section 4b, 7b, and at 
the end of section 13. The work is checked at pivotal points as it 
progresses (shown by VY). 

In the left-most entry of Eq. IV is found the minimized value of 
g*, The comparison of this value of ¢2 with that calculated by summing 
the weighted squares of the individual residuals Ve and Vy gives a 
check on the entire process, from table 1 clear through to the end 
(Whittaker and Robinson, Calculus of Observations, Art. 118). The 
check, however, is not very sensitive, 

The reciprocal matrix is calculated in Eqs. 11, 12, and 13, in 
the columns headed C,, Cz, Cg; it contains the variance and product 
variance coefficients for the parameters a, b, ¢; also is useful as a 


multiplier for solving the normal equations with any constant ("1") 
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column whatever, as has been known for many years;* see the exercises 
following, also the numerical illustration in section £30 SoLt sis Moet 
generally known, though, that the "reciprocal solution" found by using 
the reciprocal matrix as a multiplier is a very sensitive indicator of 
instability, and that it therefore breaks down in the case of near 
indeterminacy**--a fact that detracts rather drastically from its use- 
fulness in the solution of normal equations in curve fitting, where. 
near indeterminacy is surprisingly common. In extreme cases it can be 
recognized by the "freezing" of the solution--the near vanishing of the 
last coefficient (the here nee entry [cc.2] in Eq. III in the case of 

| three parameters); also by a very small value of A, the determinant of 
the coefficients, and the correspondingly large numbers found in nee 


reciprocal matrix, .(For an easy evaluation of the determinant of the 





| coefficients, see exercise 1 following). Further discussion occurs in 


example 1 of section 207 1 


I, [ab] for i 
L 


Foals 
! The short-hand notation [aa] for : = 











> 





FoF 
iz foo] for | al 
| qi 








, etc, will be introduced here for convenience in 
| writing. The normal equations 88, p. 87, with the unit matrix in 


the C,, Cz, Cz coleimns, are solved according to the scheme on page.107, 


which comes with modifications from Leland's Practical Least Squares, 





> 
me mee ee ee em em es me ae me ee ee ee me ee ee ee ee ee ee ee ee ee a a we eee ee 





The earliest reference that I know of on the use of the reciprocal 
Matrix as a multiplier for solving a set of linear equations is Art. 8 
of Gauss! Supplementum Theoriae Combinationis Erroribus Minimis 
Obnoxiae (Gottingen, 1826; Werke, vol. 4). 

** W. Edwards Deming, Science, 7th May 1937. 
* See also A. de Forest Palmer Theory of Measurements (McGraw-Hill, 
obey p. 77. 
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and is similar to Doolittle's* solution, which in turn goes back to 
Gauss**, The Gauss symbols [bb.1], [ce.2], etc., seen in Eqs. II and 
III will facilitate reference to certain entries later on, as in the 
exercises beginning on page 108, All entries below Eq, 4 arise by 
the simple operations of multiplication and addition indicated under 


"How obtained", The check marks show the "sum check" at the pivotal 





points, For numerical examples, see sections 12b and 20, Note that 


Va is eliminated in Eq. II; v, and v, are both eliminated in Eqs. LEEe | 


Eq. 12 comes by dividing III through by [cc.2] to get vg. 
7 The 

Eq. 12 comes by substituting from 11 into II to get vp. "back 
solution" 

Eq. 13 comes by substituting from 11 and 12 into I to get v,. | 





Other methods of solution have been contrived, but a complete | 
summary is not called for here. Some of them are devices for calculat- | 
ing the reciprocal matrix to be used as a multiplier (pp. 105 and 111), 
for example, T. Smith'st, and a very promising scheme of matrix squaring 
devised by Hotelling and Girshick on the basis of a theorem regarding the 
characteristic equation of a determinant, and now being critically | 
tested by Mr. Girshick in the Bureau of Home Economics. In another direc=- 
tion there is Kelley ana Salisbury's’~ ingenious acceleration of an itera- 
tive process usually known as Seidel's (1874) though described carlier 
by Gauss and Jacobi®, the same being particularly effective when the num- 
ber of normel equations is large. Then there is a fascinating pivotal \ 
* WM. H. Doolittle, Coast and Geodetic Survey Report for 1878 (Washing- 
ton), App. 8, pp. 115-118. 


** Gauss, Supplementum Theoriae Combinationis (cited on page 36) Art. 13. 


+ T, Smith, "The calculation of determinants and their minors", Phil]. 
Mag. (London) 3, 1007-9, 1927, 
































++ Truman L, Kelley and Frank S, Salisbury, J. Amer. Stat. Assoc,” laa 
282-292, 1926. sad : Ay 


© Whittaker and Robinson, Calculus of Bhaarveti one (Blackie & Son, 1924) 
Art. 130. See also pp. 30-32 of Harold Hotslling's "Analysis of a 
complex of statistical variables into principle components" (Warick & 


(Cont'd on p. 108) 
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The normal equations and their solution 
no,’ ve Vy Vo = 1 hye eee Rlutehs ) schitia 
ee eres meee ker eRe RR eit eine Cen he Pe rt ea Te a ae 
i [aa] [ab] [ae] [ao] 1 0 0 wig aye 
2 Lob] foe ] [bo ] @) 1 ») Area 3 
3 [ec } [co] 0 O i) eee 

How : 

4 obtained Loo] 0 0 0 spear 9 





[ab]* [ac][ab] _[aolfab] [ab] 





ee ay ee) ye Re 0 © ae 
fae] [aa] faajene (sad 
| ; : [abj 
ae as, [Gog “ibe? ) Lbo.l1] -=-— 1 0 v 
| | faa] 
6 Ix-[ac]/faa] oh ae Grieg se 
ix pe.1j/{ob.1] : 0 : 
on oo 16+) 7 [ec.2] [eo.g] Re lesen. i arte 


8 Ix-[ao]/[aa] 


9 IIx-[bo.1]/[bb.1] 


Ooulitx -[bo.2) /[be.2] 


IV hs hahah” Ea aed WO 


ao I solved for vz 
Le II solved for vy 


welt solved for Ve 
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iF 

y 

| 


process invented by A. C, Aitken* of Edinburgh in 1932, after T. Smith's 
method; he has now, however, superseded this solution by the intreduction ; 
of a number of important refinoments, as I have had the privilege cf | 
seeing. Mr. Girshick is testing also a new scheme originated by Professor jf 
Hotelling for accelerating the convergence of his iterative process 
described in 1933 (reference beginning at the bottom of page 106). It 
is.also interesting to note that electrical circuit machines, capable of i 
solving something like 10 linear equetions, practically instantencously | 
‘after plugsing the coefficients, are in operation and undergoing further i 
development nt several esntres, 


Exercise 1. (a) The determinant of the coefficients of the normal | 


equations on page 107 can be evaluated as follows: " 





laa} [ab] [acy 
A= [ab | too} = fbeq awl ga pkeulb hdd *anveee en (95) 


Pec] / the] [ec] | 


which is to say that the determinant A is the product of the left- 
most numbers in Eqs. I, II, amd III. This is important in consider- 
ations of near-indeterminacy. 

(b) Show that none of the left-most entries in Eqs. I, II, or III 


can be negative. 


Exercise 2, The matrix reciprocal to A can be denoted by 












































C44 Cig C33 
-41 
ER ae Co1 Cee Cos (96) 
Ca1 Cae Caz 


(a) Show that the solution of the normal equations with the 


constant columns C,, C,, and Cz leads to the values 


~—e een nn een sr a wa ee ee ee ae ee ne ae we we a a ee ee ee en ee ee 


York, Baltimore, 1933; originally published in the J, Educational 
Psychology, Sept. and Oct, 1933), 

* A. C, Aitken, "On-the evaluation of determinants, the formation of 
their adjugates,...," Proc, Edinburgh Math, SOC 0, eOl eld ooo 
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| o.. = cofactor of [aa] cofactor of [ab] _ cofactor of [ac] 
| SAE vam Cie = Cig ST noeemermnrneneny nnRNRnRERND 
A ’ A ’ A 
_ cofactor of [ab] cofactor of [bb] cofactor of [bc] 
kere A cate iy oes = SMEAR aL Gat SOS Ser sow) gece, Wo 
‘ _ cofactor of [ac] cofactor of [bc] cofactor of [cc] 
Se tes gate Cad = ao 


(Since A falls in the denominator of each slement, a small value of A, 
near-indeterminacy, results in high S. Es, of the parameters; see 


exercises 2a and 11lb' of section 19) 


(b) Like the determinant A of the coefficients, the reciprocal 


Prete ie avinetricgl i. G, Cio = Cp. Cis = Car, Cag = Cale! 


(c) The matrices A and A~* are also alike in another respect-- 


the terms on the main diagonal will always be positive. 





(ad) Show that ¢,, = 1/L[cc.2] = the reciprocal of the coefficient 





of the 3d unknown in TIT. 


Exercise 3. (a) Combine Eqs. 65 and 87 to get 





D® = [00 - [ao]lv, - [bo]vy - [colve 


. 
* 





exhibited above is actually ¢®. 


(c) Show also, by noting how Eqs. 8, 9, and 10 are formed, that 


ee 


: (b) Prove that the left-most entry in Eq. IV of the solution 
yy os leak 2 
g2 = [00] - [aalv— - [bb.1]vp - [ee.2]ve" 
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where v, = [ao]/[aa] = the value of va that would be obtained if b and 
e were fixed (not adjustable) at the values by and c,, and wherein also 
Vp = [bo.1}/[bb.1] = the value of Vp, that would be obtained if ¢ were 


fixed at the value cy, but a and b both adjustable. 


Remark 1, This result sheds a singular elegance on the form of 
solution exhibited on page 107. The term [oo] seen in Eq. 4 is the sum 
of the weighted squares of the residuals calculated under the assumption 
that a = a), b = bo, ¢ = Co (see exercise 3 of section 15, p. 97). 

The three negative terms in Eqs. 8, 9, and 10 on page 107 are precisely 


the amounts subtracted from [oo] at the bottom of the preceding page, and in 


the same order, That is to say, by the routine solution outlined on page 107 


there will appear (1°) in Eq. 8 the reduction in weighted squares that is 
brought about by allowing a to be adjustable while b and ec are fixed at 

bo and Co; (2°) in Eq. 9 the further reduction that is accomplished by 
allowing b to be adjustable while ec is held at e,; and (3°) in Eq. 10 the 
final reduction that comes from allowing c to be adjusteble, the net result 
being the minimized sum of weighted squares, ¢*. 


After a solution has been carried out upon the parameters a, b, c, 
the question often arises, what would have been the result for ¢* if the 
parameter c had not been adjusted, but had been fixed at (say) yo? Now if 
this yo is not too far from the final value of c, one need only add 
[cc.2] (c-y,)* to d® in order to see what would have been obtained for 
g* had ec been fixed at yo (see examples 1 and 2 of section 20). The value 
of o®(ext) would then be g* + [cc.2](c-y,)*® divided by n-2, not n-3 
(n = the number of points). 


Under certain conditions, the restriction that y, and ec be not far 
apart can be removed; the polynomial y = a + bx + ex* with x free of error 
is an example. It all depends on whether the parameter c enters the L 
factors of tables 1 and 2 in section 16; if it does not, then no matter 
how wide the disparity between c and yo, the term [cec.2](c-y,)*® still 
represents the increment in ¢* that would be brought about by adjusting a 
and b to the condition ¢ = yy. 


In like manner, and under similar restrictions, a term (b-B,)?/cee 
will represent the increment in ¢* that would be brought about by ad- 
justing a and c to the condition b = By (see exercise 2d). 


Similarly, the two terms [bb.1] (b-B,)*® and-[ce.2](c-y.)* added 
to the g* found in IV will give what would have been obtained for s* 
if only a had been adjusted, b and ec fixed at By and y,. The same result 
comes by subtracting [aa](a-a,)* from [oo]. .In this circumstance, 
o£(ext) would be computed with n-l degrees of freedom, 
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It is important, as a practical matter, to note that the coeffi- 
cients [bb.1] and [ec.2] needed for these increments are already at 
hand, numerically, in Eqs. II and III in the finished solution, 


Remark 2 In both parts (a) and (¢), #2 is shown as three 
terms subtracted from [oo]. Evidently 


[aolvg + [bolvy + [colvg = [aalvg’ + [bb.1]vt® + Eosee ius 


The thres terms of the right-hand member have individual Significances, 
already discussed, The three terms on the left, however, so far as I 
sce, have only collective significance; the three subtracted from [oo] 
give *, but no one term represents a sum of weighted squares removed 
by a alone, b alone, or c alone, It might be helpful to turn to 
example 3 of section 20 at this point, 


Exercise 4, Prove that the solution for Va> Vp, and v, found from 


the "1" column will also be given by the equations 


Taco aoe ant bbojes, + [eolexs 
Te ele jcgmt | bolc,, + Lcojc,, (97) 
Ve =sedlGw stu | bOlC,. + Leole.. 


This method of finding the unknowns v,, Vy, and v, is called the 
reciprocal solution because the reciprocal matrix is used as a multi- 
plier along with the constant ("1") column [ao], [bo], [co]. The 
reciprocal solution is particularly useful when one has the same coef- 
ficients, hence the same reciprocal matrix, repeated over and over 
from one problem to another, but with a new constant column for each 
problem, and hence with a new set of values for Var Vp, and Vg each 
time. See, however, the eke to aifticuities encountered in 


near indeterminacy, mentioned earlier in this section, also in example 


_ lof section 20, Theoretically, the direct and reciprocal solutions 
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should agree, and they will if the computer carries cnough decimals. 


——————— 


Bxercise 5, In matrix notation, ths results of the last exercise are 
expressed by saying that we have 

ee H 
-H being the matrix of the "1" column, and that we desire to find the 


matrix v. The solution is evidently | 


v= AH | 
To get A7* we set | 
Ac =1 
and solve for c, getting 
(ee a 


Having now the matrix A™+, we use it as a multiplier with H to find 


the matrix v from the relation above, getting v = cH, This is the 





























matrix expression for the results stated in the preceding cxercise, 


For illustrations see section 12d and example 1 of section 20, 


Exercise 6 Prove that the values of the determinants A and A~* 


are reciprocals. 

















Exercise 7, If x = rcos 6, y =r sin @, the two Jacobians as matrices 








ax ay or 98 

dr dr ox Ox 
and 

obra or 86 

aé dé oy oy 

are reciprocals of one another; i.,e., their product gives the unit matrix 

es ean | 
fare ts 
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18, The weights of the parameters; their standard crrors., The 
UA FLL LULU PLU LIU 


standard error of a function of the parameters. The standard error 
Wat 


DWAR Anil UU ULV 


of a curve. The confidence belts associated with a curve. It isa 
MADRAS IVINS ARRAN nnnnnnnnnnnnnnannnannn 


fact* that the reciprocals of the weights of the parameters are found 


on the diagonal of A™* (sce exercise 2 of the preceding section), i.e. 


Wate Lfeys, Wh = 1/¢es, Wao 1/e33 (98) 


Then since weights are reciprocals of variance coefficients (p. 12), 


Seed see Ce,0 5) | OF = c,,0* (99) 


(8. Boor la)*) = ¢.70* 


( " pb)® = ¢,,07 (100) 


-_— 
3 
oO 
law 
t 
oO 
a 
S 
ie} 
tay} 


Let f be a function of the parameters. Then 


Pena heat leapt Dyt FofpracIa%a) 


r SS Bstha + 2 fpyteTye pH %e 
+ (f,0,)? (by Eq. 47, 
Sees 


o# bee tUerCaetel hit) AC t,t. 
+ Coeff + 2 Cesfpf, 


+ 2 Coehet (101) 


As in section 6c we write for the unbiased estimate of o* by 


external consistency, 


o8(ext) = g*/k where k=n-p 
* The theory of all this goes back to Gauss, Theoria Combinationis 
(cited in section 6c), Art. 21. An excellent.reference is to Whittaker 
and Robinson's Calculus of Observations Arts, 121-123. 
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n being the number of points and p the number of adjustable parameters. 
When o is not known from any better source, this estimate may have to 


suffice, and o*(ext) would replace o® in Eqs. 100 and 101, giving 


respectively 
(Rst'd oSe Bt ofee)*t= c ,,08 (ext) 
(ore : " b)®) = e,,0"(ext) (102) 
ee " " c)® = c3,0%(ext) 

and 


(Bat'd SiS, of £)®* = o*(ext) {ei.f, + 20iel ty Perea eee 
+ 2epef? 00 eceee ee 


+ 2¢3f2} (103) 


A convenient reference showing the application of this formula to 
curve fitting is a papsr by Henry Schultz, J. Amer. Stat. Assoc. 26, 
139-185, 1930, Shultz shows curves and confidence bends of width twice 
the S. E. of the curve for several kinds of curves. It should be men= 
tioned, as Schultz does, that all these things were well known to 
Gauss and others in his time, but that they did not take the trouble 


to write out the formulas explicitly and draw the graphs for all the 





things that interest us today. It is a fact that of all the methods 
of curve fitting that have been devised, least squares is the only one 
for which the calculus of probabilities has been successfullyeappliecd 
toward the calculation of chances associated with the parameters and 
functions of them. The usefulness of Eqs. 102 or 103 or any like them 


depends only on whether the assumption of normally distributed observa- 
tions is well enough obeyed, not on the intermediate steps of calcula- 


tion, which are mathematical, and save for blunders, above reproach, 





ae 
< 
i 
, 7 
vie | 
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The student who has progressed thus far in the theory of least 
squares is strongly urged to study Ch. 5 of R. A. Fisher's Statistical 
Methods for Research Workers, wherein examples of the manipulation of 
the reciprocal matrix and the testing of parameters will be found, 
masterfully exhibited, 
Cnly a brief discussion will be given here, When we write 
Va=ps bea; b,c) (104) 
and ask for the S, E. of y, we are merely asking for the S. E&. of a 
function of a, b, and c, but not of x; consequently, we can apply 
gs. 101 or 103 at once, This is what Schultz has done in his paper, 


x enters merely as a constant, 


Confidence bands are of 
two kinds, (i) o known, the 
normal type; (ii) o estimated, 
the Student type. To construct 
a 50 percent confidence band of 
the Student type we look up tso5 
in Fisher's table of t for n-p 


degrees of freedom,* then compute 





[Y-y| for several values of x, ; 


using the equation 


Figculiy VA Curve and: 
(105) confidence band 


: aise 
72 ~ Esta Soeiwn ot 


Se RS AON eR See ee ee ey Sle ee ma in ene Ses eae SS See eal SED ONS dele SD Ooi OD eave ET cote IE wey eae SED OP mag Oe Uae Oem MSS em ce ce coms wie mate Sih we ow pote wits wie se ey bs a ns eae at 





* V. A. Nekrasseff's very handy nomograph may be used in lieu cof 
Fisher's tatle of t; it was published in Metron SUNG. Sy (L950 ane 
is reproduced in W. ge Shewhart's Economic Control of Quality (ven 


Nostrand, 1931) p. 490; see also Deming and “Birge Statistical Theory 


of Errors, p. 136. 
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The distance |y-y| laid off above and below the curve defines points 
on the confidence band. A 95 percent confidence band would be com- 
puted in the same way, but with ts, replaced by t,,5; see Fig. 13 en 
page 168 for an example. (The capital Y used here is not to be con- 
fused with the same letter used in Fig. 9 and elsewhere for an observed 
coordinate). 

The first type of confidence interval, the normal type, is con- 
structed the same way, except that o being known, the S. E. of y is 
known and not estimated, and the normal integral is used in place of the 
Student or t integral. 

It must be remembered that a new set of points will give a new 
curve (i. e., a new set of parameters), hence if the confidence band 
is the normal type, the whole thing, curve and band, will be shifted 
to a new position by a new set of data. Moreover, in the Student type, 
the estimate of o will fluctuate from one set of data to another; then 
not only will the curve and band be shifted to a new position by a new 
‘set of data, but the width of the band itself will also be different. 
The researches of Walter A, Shewhart have an important bearing on such 
matters; his Washington lectures (liarch 1938), to be published by the 
Graduate School, should be consulted. The reader is also referred 


to Neymen'ts Washington lectures (cited on page 18), pages 143-160, and 


to Fig. 11 on page 141 of Deming and Birge's Stetistical Theory of Errors 


(cited on page 20). 
Confidence intervals for any other function of a, b, and ¢ are 


made up in like manner, 
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19. Exercises and notes on tho formation of normal equations 
PAA GUUS UVP PII 


VU 


for various functions. The following examples are given as exercises 
for two purposes; first, to provide practice in setting up normal 
equations anc gaining familiarity with the procedure in some common 
formulas; and second, in order to have the results for ready reference, 
Once these exercises are mastered, other forms of functions arising in 


practice should cause no cirriculties, 


eo ee 


A special note should be made concerning the fitting of poly- 
Peres yee) + bx) a+ bx + cx*®, and the’ like x free of error. 
Some of the following exercises deal with this type of formula. Space 
will not be taken up here for a study of certain short-cuts that apply 
in the fitting of polynomials to equally spaced points (i.e. equally 
spaced along x), x free of error. To dismiss ths subject with a refer- 
ence, I shall mention what seems to be the most promising paper on the 
subject: Raymond T. Birge and John D. Shea "A rapid method for calcu- 
lating the least squares solution of a polynomial of any degree", 


University of California Publications in Mathematics, vol. 2, No. 5, 1927 
unfortunately now out of print. There is a strong possibility, however, 


that Professor Birge may publish a short account of the method with 
examples in the J, American Statistical Association for 1938 or 1939, 


In this paper, Birge and Shea give explicit directions, tables, 
and examples for the fitting of polynomials. A feature of the method 
is its rapidity and ease, owing to the small number of figures required 
in the solution. Up to a certain point it seems to be equivalent to a 
method advocated by Harold T, Davis, but beyond that point the remain- 
ing steps are actually much simpler than Davis', and require only 
about half as many decimals. Any one having a great deal of work to 
do in the fitting of polynomials should become familiar with Birge and 
Shea's method, It is, of course, equivalent tco--i.e. gives the same 
results as--any linger method that is truly a least squares solution, 


Of late there have been many papers on the fitting of polynomials 
from the standpoint of orthogonal functions. The reader may wish to 
consult articles by A. ©. Aitken, Proc, Royal Soc, Edinburgh 53; 

54-78, 1932-33; also Max Sasuly's book Trend Analysis (The Brookings 
Institution, Washington, 1934), -Another important contribution, using 
summation methods, is a paper by Frederick F, Stephan, J, Amer, Stat. 
Assoc, £7, 413-423, 1952. 


’ 


= 
ae 


-118- 
(Sec. 19, 


Ex, 1,the line] 


Many people will prefer the form of solution given by R, A. Fisher 
in sections 28, 28,1 and 29.2 of the 6th edition of his Statistical 
Methods for Research Workers (1937). Miss Day tells me also that in 
the Forest Service, the method of Lorenz has bean found very useful and 
easily learned; the reference is to Paul Lorenz Der Trend (Published for 
the Institut fur Konjunkturforschung by Reimar Hobbing, Berlin S.W, 61, 
1931). The copies are marked "Bezugpreis des Sonderhefts 14,40 RM," 


THE LINE* 








Exercise 1.(a) Given the line 
y= a * bx 


to be fitted to n points, x free of error, all y coordinates of equal 
weight (unity). Here we take 


F = y - (a + bx) 


The derivatives are Bin Pah By ee Xs 
FF eb (not needed here), Fy =] 
Live L.A Why? +See Bar 85 ps: 85) 


With the approximate values ap and by, we compute 
Ba by Wee Caeut Dok! 


at every point. Since L = 1 at every point, tables 1 and 2 of section 
16 coalesce, and the normal equations are seen to be 





No. Va Vp = 1 : Ce op Sum 
wy n [x] -(Fo] cl ~"C AA 
2 is") -[xF,] © if ae (Set 1, 
Exercise 


3 Ri ©) 0 lite if 


[x] means >x, [F,F,]means the sum of the squares of Fo, etc, 


* The line y = bx, forced to pass chxvoueh the origin (i.e. with aie 
was discussed to somes extent in section 7, “ ; 
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Exily othe Line! 


(b) The solution for V, and v, found by the routine of section 
1? or any other method of solution, is 


Tee ra fn ne ar 
vy = - {{xFo] + x{FoJ}/m, 
where Ms = [x®] - nx? 


NUg is the second moment of the x coordinates about an axis parallel 
to Oy and passing through the centroid x, jy. 


(c) The adjusted values of a and b turn out to be 
& = 8) - Va = y - bx 
b= Do - Vy = X(x-x)(y-¥)/nte = ([xy] - nxy)/nu, 


The fitted line therefore passes through the centroid x, y. But note 

that when there is error in both x and y coordinates at some or all of 
the observed points, the weights being such that wx/Wy is not corstant 
throughout, the line does not pass through the centroid, see Remark 1 

in Bx. °4. ' 


(ad) The solution just found for a and b is the same as would 
have been found from the normal equations 





No. 8 heres 1 C, Ce Sum 
I n [x] Cy] i s) ‘ 
: (Set 2, 
2 Lx? J [xy ] 0 J. Exercise 1) 
3 Ly*] 2) 0 " 


That is to say, it is permissible here to take a,j and bo both as zero, 
whereupon Fy, is simply yYop,. The normal equations Set <« then give a 
and b directly. f fat 


Note that F, is required at every point for Set 1, but not for 
met «=. However, in Set 1 it is’only the residuals v, and v, that are 
to be solved for, the main part of the adjustment having already been 
taken up in a, and bo; the additional accuracy required in Set 2 
necessitates more figures in the formation and solution of the equa- 
tions, the unknowns being the full values of a and b; and this will 





[Sec. 19, : | 
Ex.1-2,the line] 


fore is usually advisable to find good approximations and use Set .l. { 
The better the approximations, the fewer figures required. Birge and 
Shea make use of this principle in their method of fitting polynomials 

(mentioned in the opening paragraphs of this section). 


usually more than offset the time required for computing F,. It there- | 
| 


| 
(e) When the solution of either Set 1 or 2 is carried out ac- | 
cording to the scheme of calculation exhibited in section 17, the | 
left-most entry in Eq. III will be the minimized 2? or Lyon - Yealc)®: 
The sum of squares removed from (FoF, ] in the case of Set 1, and from 
[y#] in the case of Set 2, by the successive adjustments of a and b, 
appear in the left-most entries of Eqs. 5 and 6. Show that 


s* = [yy] - ny® - nugd? | 


the last term being the sum of squares removed by allowing the line | 
to have slope b instead of slope O--in other words, the sum of squares 
removed by regression, The two terms ny© and nu,b* appear in Eqs. 5 


and 6 of the solution of Set 2. 


(f) If V denotes Yoos ~ Yeate 2t any point, the solution for a 
and b renders 7V = 0. (But note that neither 2V nor ZwV -is, in 
general, zero in least squares solutions; it only happens to be so 
here. In fact, in section 7a we saw a simple example wherein neither 
2V nor 2wV was zero. See remark 3 in exercise 4; also exercise 5), 


Exercise 2. (a) The reciprocal matrix for the normal equations in the | 
preceding exercise appears in the C, and C, columns of Has. 7 and 8 
(these numbers refer to the solution of the equations solved according 














to the form in section 17). It turns out to be § | 
L x2 bi x ' 
n Me Ne 
Aq? = 
DONE i 7 | 
Ne Ns 


(b) Hence the weight of a is n/(1+x’ Aue), and the weight of b is 
Ne (see section 18). Our confidence in b thus increases as the "spread" 
of the points increases, Is this reasonable? Why does the weight of 
a depend on x?° . fa 
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Bx,.2-3, the line} 
(S.E, of a)® = o2(1/n + x2/nu,) 


(S.B. of b)® = o2/ni, 


i 


Note that the weights and S.Es. of a and b do not involve the y coordi- 
nates of the points. Compare with part (d) of the next exercise, 
Note also that 

ce et 

nMUbe = Ae 


[x] [x*] 


hence near-indeterminacy (small A) is closely associated with high 
o,6bs, of a and b, and a rapid "fanning out" of the S.E. of Vaote 
each side of x, y (sse the next part; also exercise 2a of section 17). 


(d) From Eq. 48, p. 27 and the reciprocal matrix of part (a), 
prove that the 


2 ; = 
(Som, of a a = i {1 + (x-x)* /us} 


Thus the S.E. of Yougjq is least at the centre of gravity (x, y) of the 


points, and fans out each side of it. See section 18 and reference to 
Menry oChuULLZ: Blsowkies, ll and 12, 


(e) The S.E. of the calculated line of Ex, 1 at the centre of 
gravity is of/n, as it would be for n observations made on a single 
unknown. (Do this in two ways: 1° put x = XK in part d; 2° put x = 0 
Peery oO for the SiR, of 'a) | 


Exercise 3. (a) Carry out the solution of the normal equations of 
Ex. la in symbols, following the outline given in section 17, and 
show that the minimized value of ¢* or of Ul¥ong - Yeale)® comes in 
the "1" column of Eq, III (which will be the left-most entry in III). 
The same is true if the approximations a, and by are used, as is 
advised in Ex. ld. 


(b) Show that this can be written 

res” = n(1-r* )s% 
r= 2(x-x) (y-y)/nsxsy = the correlation coefficient 
and nsy =Lyy] - ny* 


ns& =[xx] - nx® 


=122-— 
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(s,£is here used in place of Up, for consistency with Sy). 


(c) The estimate of o mads from the fit of the line is (see sec~_ 
tion 6c) a 


o2 (ext) = n(i-r?)s¢/(n-€) 


1-ré 
n-2 





(a) The (Est'a S.E, of a)® = sf (1+x*/sz) 





© 1l-r z. \e 
’ pals 2 e 
(Est'd S.E, of b)* = => (s°/s%) 


Note that the Est'd S.Es. of a and b involve the y coordinates: compare ~ 
with part (c) of the preceding exercise. 





1-ré 


(e) The (Bet” a aha Dh Yeates = nooo 


sy {1+ (x-x)* /s2} 


Exercise 4 (a) If both x and y coordinates are subject to error with 
varying precisions at some or all of the n points, one must go through 
the calculation of tables 1 and 2 of section 16. . For the line 


a tox 


" 


y 
one takes 


F Vena a=) DX 


Some good approximate values a, and bo having been found, one can then © 
calculate the numerical values of 


Fo = Yoos - (85 + Do Xopq) 
at the n points. 
The derivatives of F are 
eT IMME eine alia’ 's Willies i bie Stk ses 
whereupon Le bea, + 1/wy 


L varies from point to point with w, or Wy. 



































ee 


(See. (294 
Ex, 4, the line] 


The headings for table 1 of section 16 would be 











ny or 


Point No, Wy Wy L VL ae? 2 Fe 


-—_—_—_————————— 





(It is not necessary to tabulate Fy, Fy, Fg, and F, since they remain 
constant from point to point) 


The headings for table 2 of section 16 are then 





h, or hintaan eee co 
Point No, F,/VL ey TE FLL = x//L FLA Sum 








It has already been remarked that there is some theoretical 
advantage in writing W in place of 1/L, though it is a fact that with 
machines having automatic division and two dials for quotients--one 
for the individual quotients needed for table 2, and another for cumu- 
lating the quotients across the rows for the "Sum" column of table 2-- 
there is a practical advantage in tabulating VL rather than /’W in 


table 1, and using divisions by VL, rather than multiplications by 
YW, to form table 2, 


Writing now 
L/W = b2/wy + 1/wy 
we see that Y= w. if x is free of error or if b= 0 (see the next 
exercise), and W = w,/b® if y is free of error, but that both terms 


are required if x and y are both subject to varying errors (see Hx. 8b). 


In terms of W the headings of table 2 would be 


ee 





1s Past ey 
Point No. VW xi VWE 6 Sum 


—— re 


| (b) The normal equations are formed in the usual way by sum- 
ming squares and cross~products from table 2; they can be symbolized as 


No. Va Vp = sik. a Pw ts Co Sum 
I tw [Wx] -[WFo] ae 0 - 
! (fo : (Set 1 
2 =! th di 3 
2 , (Wx J (WxF,] 0 if pea naea 
3 (WF .Fo] 0 0 - 


- ig 
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Ex, 4, the line] 


The solution of the normal equations gives Vg and vp from the "1" 
column, and the reciprocal matrix A7* as usual from the C, and Cz, col- ) 
umns. The adjusted values of a and b will be Ta 


A= 8 - Va 
b ® Do. - Vp 


The solution gives the minimized value of 2(w,V¥ + WyV¥) in Ba, 
III, column "1". That portion of the sum of the weighted squares 
subtracted from [WFoFo] by shifting the first parameter from aj to a | 
appears just above in Eq. 5, and the portion further removed by shift- 
ing the second parameter from bo to b appears in Eq. 6 {see exercise 3 
of section 17; also exercise 1 of this section). ; 


Note that a, and bo may be taken as O (with the necessary in- 


crease in the number of decimals required in the normal equations), 
so far as F, is concerned, in which event F, becomes simply 


Fo = Yobs 


The normal equations are then written 


No. a b = 1 CN. Ce Sum 
I Cw] [wx] +(Wy] 1 0 . 
: : (Set 2, 
2 [Wx*] +[wWxy J 0 1 = Exereise 4) | 
Z (wy 2] om 0 = ; 


and the solution gives a and b directly. Why are more decimals re- 
quired in these equations than in the preceding ones giving the 
(supposedly small) residuals v, and v,? 

But note carefully that an approximate value of b must be used 
in the calculation of W at each point where x is subject to error, 
bo may be called O in the calculation of fF, (as noted above) but not 
in the calculation of W. If this admonition is disregarded, the ef- | 
fect of the weighting of x is lost, In fact, if it. turns out, wnat 
the approximate value of b used for calculating W was too far removed 
from the final b, it may be desirable to make a second adjustment to 
secure improved weightings W, which can be obtained by using the value 
of b from the first adjustment; but this is seldom found necessary in 
practice,* 


a a re ee ee ee ee ee ee a a a a a i a a a ee ewe ee 


* As Gauss put it, in a somewhat different problem: "Quodsi dein 
calculo absoluto contra exspectationem valores incognitarum p'’, q', 
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Of course, it may happen in some particular problem that b ac- 
tually is very small, and that O is therefore a good approximation for 
b. In such circumstances the line is practically horizontal, and the 
weighting of x does not matter much, and the computer may as well 
simplify matters and set W = Wy» ignoring the weighting of x--not 
because x is free of srror (i.8. not because w, is infinite), but 
because b is zero or nearly so. The student should ponder over the 
Situation where b is actually known to be 0; do the values of x count 
at all in the solution? Does this not take us back to "the simplest 
problem in least squares" of section 3, the x and y axes being inter- 
changed? The solution would be simply 








a = dwy/ow 


and the weight of a would be 2w, its S.E. o Sw (w being written for 


wy for convenience), 


Remark 1 When x and y are both subject to error at some or all 


of the observed points, the line does not pass through the centre of 
gravity 


% = Cx, J/lv,)],  ¥ = Lywyl/Lwy] 
But the line will in any case pass through a quasi centre defined as 


= [xw]/Cw] , y' =Lywl/(w] 


Remark 2 With L/W Written, in place. of L,.Eq..85,.p. 85. gives 


Ae i PYF, fe PyFy 
W Wx Wy 


As has already been seen, the first term drops out if x is free of 
error, and the second term drops out if y is free of error. To make 
the change from a solution in which x is free of error to one wherein 
both coordinates are subject to error, we merely add the other term in 
1/W and recalculate W at every point, the procedure being otherwise 
the same. There is a close analogy with celestial mechanics; when one 
wishes to conpute the orbit of a body of mass m about another of mass 
M, he may at first make the simplifying assumption that M is infinite 


ee ee ee ee me a ee ea ae a ae a a a a a a ee eee 





r', s' etc. tanti emergerent, ut parum tutum videatur, quadrata 

Pe aieraaia neglexisse, eiusdem operationis repititio (acceptis loco 
Poesrumaree ype 0 .eLc, valoribus, correctis ipsarum ip, q, r,s; 
ete.) remedium promtum afferet." Theoria Motus Corporum Coelestium 
(Hamburg, 1809) Art. 180. 
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(i.e. immovable), and solve the equations, later replacing m by U 
where 


This replacement yields the absolute motion of the two bodies, neither 
being of infinite mass (i.e. neither one immovable). 


Remark 3 When x and y are both subject to error at some or all 
of the points, we can not always assert that 








Een Mores v= Owen wi WyVy) = 0 


though these may sometimes happen, as in Exs. 1 and 5, q. v. We have 
already seen a simple example in section 7a where these summations 
were not zero, There is, however, a property of least squares by 
which one can always assert that after the adjustment, 


(For definitions of U,, ete., see Figs. 8 and 9 on pp. 82 and 83, This 
property of least squares was published by me in the Phil, Mag. 19, 
389-402, 1935). 


Remark 4, It is interesting to note that in the routine solution 
of Set 2, the minimized 2* appears in the left-most entry of Eq. III, 


but that, in contrast with Sct 1, unless the final value of b is actually 


O or very near, the entry in Eq. 6 directly above ¢* will not show the 
increment in ¢* that would result from fixing b at either the value 0 
or bo. The reason is that a good value of b must be used in W at each 
point where x is subject to error: if we want to know what the solution 
would have been with b = 0, we must actually make a solution with b 

set equal to 0 in the computation of W, in which circumstance W reduces 
to Wy, as already noted. . 


Exercise 5. Given 

yo sates +h 
to be fitted to n points, x free of error, the y coordinate having 4 
Weight w varying from point to point. This is similar to Ex, 1 ex- 


’ 
cept thay now the y coordinates have unequal precisions. Here we take 


Pomcyo-la; —) bE 
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as in the preceding sxercisc, the derivatives being also the same, 

| But since x is free of error, w, is infinite and it follows that W 
(defined as 1/L) is none other than w,. All we have to do is replace 
| W in the preceding exercise by w,, and the results will apply here. 
The headings of table 1 of section 16 will be 





me LOY 
Point No. ay viWy Sum te a he 














For table 2 





nH, or s 
: Re ete Vly FL/VL = -x/wy VwyFo’> Sum 





The normal equations are written in the same symbols as those of the 

I preceding exercise, but with w, in place of W. Eq. III in the solu-. 

tion of the normal equations gives the minimized value of Xwy(Yops - Veale! 
In Eqs. 5 and 6 ars found the portions of the weighted squares removed 

by a and b, as in exercise le, 





| Note that as in the precedin: exercise, it is permissible to 
take a, and bo as zeros, if the number of decimals in the normal 
equations is increased accordingly. In this event, 


Mae. Uohs 


and the normal equations may be written 


No. a by eegaal Gs Cs Sun 

I Lw] [wx] wes 1 0 Pee ore! 
2 [wxx] [wxy J 0 1 : 
2 [wyy J 0 0 - 





giving 4 and b directly. kq. III will give the minimized value of 

2Wy(Yobs - Yealc)* in the "1" column and Eqs. 6 and 6 will show the 
sum of squares removed successively by a and b, as in exercise le; 

since wy is infinite, the question of an approximate values of b for 
use in the calculation of W does not come up. 





Remark For the conditions stated (x free of error) the sum 
of the weighted y-residuals is zero, i.e. 


2WwV = 0 
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(See remark 3 of exercise 4) 


Exercise 6, (a) Given the line 
y=a+ bx 
to be fitted to n points, both x and fy coordinates subject to error but 


in such a way that w,/w, is constant and not infinite nor zero, the 
line passes through the centre of gravity x = [w,x]/[wy], ¥ = [wyy]/Lwy), 


with slope 
_ clwv®] - [wu?] + Vi{elwv2] - [wu2]}% + 4clwuv* 


eclwuv # 
(This is equivalent to a result obtained by Kummell in 1876, Karl Pearson 
in 1901, and Gini in 1921; references in section 1). Here u and v are 
the x ane y coordinates of a point, measured from the center of gravity 


Rey Syemil = ae x, and v3 = ¥; - y. c¢ is written for Wy/ We, and 
w in place of w,, for conrentenca. 


(b) If the plus sign be changed to minus in front of the radical, 
the result is the slope of the worst fitting line, that which maximizes 


the value of Z(wxV,* + wy Vy imine 


(c) Prove that under these conditions of weighting, the best and 
worst fitting lines are perpendicular to each other, 


Exercise 7. (a) Given 
y= a+ bx 


to be fitted to n points when y is free of error and all x coordinates 
are of equal weight (unity), we may write 


= Pat OF 


and find the following normal equations for p and q. 





No, Pp q = 1 Pi OF Sum 
I n Ly] [x] 0 - 
2 Lyy] Cyx] 0 - 















































— ——— 
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cates are like Set 2 of exercise 1 with x and y interchanged, 


‘Eq. III in the solution. of the normal equations gives 2, res® 


.whsre now the deviations are measured parallel to the x-axis. 


(b) The reciprocal matrix A™~ found in the C, and Cy, columns of 
Eqs. 7 and 8 of the solution will be 


1/n + y?/nsé - y/ns2 


ae J “y 
oer 2 2 
| y/nsy 1/nst 


where Sy has the same signifié@ance as in’ exercise 3, 
(c) (The S.E. of ra ae = 0*/n) Pe eee: Natal ace 


(d) The normal equations of exercise 7a give XV, = 0. 


(See remarks in exercises lf, 4, and 5). 


Exercise 8. (a) Prove that with y free of error, and all x coordinates 
of equal precision, the normal equations for a and b (or for v, and Vp) 
in exercise 4 will give the same line as the normal equations in exer- 
cise 7a (i.e. will give p = -a/b. and q = 1/b), except for the effect 

of the neglect of the squares of the x-residuels. The solution of exer- 
cise 7a is the more accurate in not throwing away any higher powers 

of residuals. This may occasionally be important. See also exercises 
18 and 22, 


(ob) Show that if x has the same weight (i.e. the same precision) 
over all n points, and y likewise, x and y both subject to error, the 
line that one gets by the exact solution given in exercise 6a lies 
between the two false lines that one gets by (1°) throwing the adjust- 
ment all on to y, using the equations of exercise 1; and (2°) throwing 
the adjustment all on to x, using the equations of exercise 7a; but that 
these two false lines differ only in the effect of the squares of the x- 
residuals and of the y-residuals, respectively. (Hint: both terms of 
1/w in exercise 4 are constant over all points when wy and wy are con- 
stant; henes, so far as the values of a and b or p and q are concerned, 
W can be put equal to unity at every point in all thres solutions--in. 
the correct solution, and in the two false solutions, The normal 
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equations of exercise 4 will then give identical results for all three, 
But the normal equations of exercise 4 can be in error at most by the 
neglect of higher powers of the residuals, hence the false solutions 1° | 
and 2° can differ from the true solution only through the neglect of such | 
terms.. This means that when x has the same weight over all points, and | 
y likewise, the false solutions will hardly be distinguishable from the 
true solution if the residuals are all fairly small.* 





Note that if W is constant from one point to another, it is advis- 
able for convenience of computation to choose the system of weighting 
so that W = 1 at all points. This is only saying that the arbitrary 
factor 2* in Eq. 13 of article 5 is tc be chosen so that W= 1. Then | 
o* in the left-most entry of Eq. III in the solution of the normal 
equations comes out in the same system, and o®(ext) = 42/(n-2) is the | 
external estimate of o# in the same units as were arbitrarily chosen 
Pore it, 





(c) All three lines of part (b) pass through the centre of gravity 
x, y (called also the centroid). > | 


Remark, Statements similar to those of ‘part (b) will hold for 
any curve when the combination of the form of the function and the 
weighting of ths coordinates causes both terms of L/W to be constant 
over all n points, Example 3 of the next section is an illustration 
in three dimensions (three terms in 1/W). | 


Exercise 9, For the line y = a + bx fitted to n points, the following 
expressions hold (all due to Karl Pearson, Phil. Mag. (London) 2, 559- 
OV. 1) 


M 


(a) res®* = n(l-r”)s® When x is free of error, the y coordi- 
Y nates all of equal weight (unity); the 

deviations measured in the vertical, 

(This result was given in a previous 


exercise). 


























i] 


(db) & res® = n(l-r?)s When y is free of error, the x coordi- 
nates all of equal weight (unity); 
the deviations measured in the hori- 


PA BUN Salley 











ce re we we a a a i a ae a a ee ee 


* This fact wes noted by me without proof in the Proc. , Physical Soc. _ 
(London) 47, p. 107, “1955; 
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Gy ireat alanis tae snflaene ine \eu4+ Are aboe 
(c) 2 res gn{s2 * Sy V(s2 Bee + 4r sect} 


When the x and y coordinates are of 
equal weight (unity), the deviations 
Aan measured perpendicular to the fitted 
line. 


In these formulas, BS) Sy and r have the meaning ascribed to them in 
exercise 3, 











THE PARABOLA 
| Exercise 10. Given 
; y # a + bx + ox® 
to be fitted to n points, x and y having weights w, and Wy at any point. 
| Here we take : 
F = y -(a + bx + cx®) 
| Fel Wapgititont Pokppg tuCoXeya) ae: Usual, 
| 
i The derivatives of F are 
FY oe -(bd he Pox). Fy = 1 
= - Sh Sige 
is ey: 7 a a x 
L-or 1/W = (b,+ 2cx)* /w,, + 1/wy 
The headings of table 1 in section 16 will be 
ee a Piere twos ¢ ol myoe Od Re Morr YaninA 
Point No,: -(b + 2cx) y 





| It is understood that in calculating all quantities under these 

i headings, x and y are to be replaced by their observed values, and a, b, 

| e; Dy a5; bo; Cy. It is not necessary to tabulate Fy and F, because 
they remain constant from point to point, : 
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The headings of table 2 will be as follows 


h, or F,/VL Fp/A Poh Fo/VL 


Point No. = -Vi! = - VIX = =VWxe * VWF 5 Sum 


a ee CS 


The age Ag process of cumulating sums of squares ae eross-products in 
table 2 yields the normal equations 








No. Va Vy Vo = ¥ Sy Ge oa Sum 
I [w] [wx] (Wx? J -(WFo] 1 ) 0 - 
2 Zt Wes -( Wy ) 
2 | [Wx® j [Wx ] (Wx 5] 0 s (set 1 
s (wx*] -[iwx*F,J 0 0 1 - 
4 -(WF Fo] 0 G 0 - 





The solution, carried out by the usual routine procedure, gives 
Va, Vp, and Va, whence the adjusted values of a, b, and c are 


rey) 
" 


The minimized value of #* or X(w,V,? + WyVy* ) will appear in ; 
Eq. IV, column "1", This will be simply 1 w,V? if y is free of error, 
and 2, wyVyF if x is free of error. Directly above, in Eq. 8, appears 
the sum of squares that is removed from [WF,F)] by shifting the-y- 
intercept from a, to a; in Eq. 9 appears the further decrease brought — 
about by allowing the second parameter to shift from by to b;-:and'in 
Eq. 10, just above ¢*%, appears the portion of the sum of SGRCR that 


is finally removed by adjusting the parabolic term from Cha to cx® 
(see (8x203 of ogee 4417), 


The reciprocal matrix A-* will appear in the C,, Ge C. columns | | 
of Eqs. 11, 12, and 13 (the "back solution"), containing the variance — 
and product -variance coefficients for a, b, and c. (See Ex, 12 for the 
matrix A71 in a special case.) 





eS mL ee a ng nao i a 


Sas 


Le a a Sy TO TONE UL A ITE III IO Et Oe RR ES SERN CR OR EE WELD 
= =e : - . s 
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Note the similarity between Sets 1 of exercises 4 and 10. 


Note that if x is free of error, a9, bo, and Cg may be taken as 
O so far as F, is concerned, in which event Fy, becomes simply 





Fo = Yobs 


The normal equations giving a, b, and c directly are then 








No, a b Cc = HK C, He on Sum 
L._ (WJ, Lax] [we 1] wy) 1 0 0 E 
e b Wx j [ wx? Wxy C l re) 
il (Set 2, 
3 (Wx*} ([Wx2y] 0 Ras ee _ _ Ex. 10) 


4 ~ [wyy] fe) 0 0 a 





More decimals will of course be required than if good values of ao, 

bo, and Cy had been used in the calculation of Fo, and the previous 
normal equations set up for Vz, Vp, and ve. Why? The reciprocal matrix 
is unchanged, and the minimized value of X(w,V + Ae again comes 


in the left-most entry of Eq. IV; but, as in exercise 4, remark 4, the 
entries directly above it in Eqs. 10 and 9 do not show the increments 
in the sum of the weighted squares that would result from setting 

c = 0 and b= c = 0, respectively. The only exceptions come when b 
and ¢ are wery small, or x free of error. Hy 


Note the similarity between Sets 2 of exercises 4 and 10, 


Note carefully that approximate values of b and c must be used. 
in the calculation of W at each point where x is subject to error, 
See the notes at the end of exercise 4, which apply here with-obvious 
modifications. 





Exercise 11.-. Given* 


yorvact bx +o exe : . es sin dental . singh: 


to be fitted to n points, x free of error, all y coordinates of equal 
weight (unity). If a5, bo, and Cg all be taken as 0, the normal 

* See the notes at the beginning of this section for special methods 
applying when x is equally spaced. 
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equations giving a, b, and e¢ directly are 





No, a b oh AG 3 ui Gy Cy Cz . Sum 
I n [x] [x®] Ly] 1 0 @) - 
2 be la ei [xy] 0 le wrone - 
é | ks et, [x2 y] 0 0 1 = 
4 i Lyy] 0 0 0 = 


i 


These arise immediately from the last set of normal equations of the 
preceding exercise by noting that under the conditions wy, = oc and w= 1 
throughout, W = 1 throughout, Eq. IV in the solution of the normal equa- 
tions will contain the minimized value of X(Yops - Yealc)* in the "1" 
column, The sum of squares successively removed by the constant, linear, 
and parabolic terms will appear in the "1" column of Eqs. 8, 9, and 10 
(see exercise 3 of section 17; also exercises 1, 4, 5, and 10 of this 
section), 


Note the similarity between these normal equations ana those of 
exercise.1, Sete. 


Exercise 12. (a) In the preceding exercise, let the origin of x be 
taken at the mean value of x, and let mu, be written for [x*] and ml, 
for [x4], Then the normal equations are 





No, a b Cc * 1 Ci Ce Cz 
tiny wah 0 NL» Cy] 1 0 0 
2 Nis ) [xy ] 0 2 0 
3 TL 4 [x€ y] 0 0 1 
Air Lyy] 0 cage 0 





Find the reciprocal matrix to be 
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Ua i jb 2 
TUL 4 -NULE 0 nh gn e 
1 | 
Avi = 0 Tle 0 
~. Mes 4 
TUL 4 ~T3 DU 4 -NL3 


This enables one to write down the S. E, of the fitted curve, or 
any function of a, b, c, in terms of o (see section 8 and exercise 2 
of this section). In particular, 


ie ee e/a) 


Wy = MU,, as he aes 
Wo F Hitt a Us) 


MHetS: ES of Veaic)’ ot the centre of gravity is equal ‘to 
o®/n(1 - uZ/t,), which exceeds the value on which was found in exer- 
cise 2 for the line. 


(b) Show that the determinant A of the coefficients is equal to 
nUs(i4 - Us); hence that near-indeterminancy (small A) will result not 
only in instability but also in-high S.Es. for a and b and rapid fanning 
out of the S. BE. of Veale (see exercise 2c; also exercise 2a of section 
i) s 
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THE EXPONENTIAL AND ITS LOGARITHMIC FORM 


Exercise 13. Given the equation 


—- 


to be fitted to n points, x free of error, all y coordinates of equal 
precision (unit weight). Here we take 


F=y- ae 
Good approximate values of a and b can usually be found by plot- 

ting log y against x: assuming that they can be obtained, we write 
Dox 


(Y denotes an observed 
y as in Fig. 9, p. 83) 


Ean” (oeeeae 


The derivatives of F are 


Peg | ere ier y/o, eal 2 ane 
W = 1 at all points, hence tables 1 and 2 of section 16 coalesce. They 
will be made up as follows, For convenience in writing, the subscript o 
will be withheld from the a and b. X,, Y,, etc., are the observed x and 
y coordinates of the n points. 


Tables 1 and 2 





h E Fy Fo Sum 
bx 
1 -Y,/a =KeX Y,-ae + ~ 
2 -Y,/a ~XoYo Yip -~ae Xe = 
ete 








nn = ee ee 


The normal equations are formed from this table by summing squares 
and cross products: 


No. Te Vy = 1 Cy er Sum 
I [xe(/an} xy? fe} -[YF,/a] f 0 ~ 
2 pers -[XYFp | 0 1 ~ 
3 ey) 0 0 - 
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The solution of the normal equations by the usual routine 
described in section 17 gives vg and vp, also the reciprocal matrix, 


and the minimized value of tres*, The adjusted valucs of a and b are 
then 


& = a, 
pees pont 


The left-most entry in Hq. III of the solution of the normal equa- 
tions gives 2#* or 2res®, the deviations all being measured in the 
vertical (i. @, parallel to Oy). Directly above ¢®, in Eq. 5, appears 
the sum of squares removed by the shift from & to a, and in Bq. 6 

ths further decrease brought about by adjusting the exponent from box 
to bx, 


Exercise 14. If in the preceding exercises, the x coordinates are free 
of error but the y coordinates havo unequal procision, designated by 
weight w (varying from point to point), W is no longer unity, but is 
equal tow, Table 2 of section 16 then runs as follows: 








h BoA Fp/VL VWE 6 Sum 

i: -Vw4¥,/a -X,Yj/w, /w,(¥,-ae>*2) - 

a ~V/WeY,/a -X,YoVWe VWe (Ys -ac Xe ) a 
eve. 


the approximate valués a, and bo being inserted for a and b, 


The normal equations, formed from table 2 in the usual manner, 
can bs symbolized as follows: 


No, Vp Vp = ue Oe Cy Sum 
I [wY2/a2] [wxy2/a] -[wYF,/a] 1 0 - 
eS [wx *Y =] -[WXYF 5] 0 iN - 


eR er ne | ne i EE RNY 1 ER ER eT cee 
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In the solution of the normal equations by the routine of section 
17, the minimized value of Zwlyops - Ycalc)* comes in the left-most 


entry of Eq? Ill: Just above, in Eqs. 5 and 6, will appear the portions 
of the weighted sums of squares removed successively by adjusting a and 
then b (see exercise 3 of section 17; also exercises 1, 4, 5, 10, ll, 
12, and 13 of this section). 

The adjusted values of a and b are, as usual, 


Ce had etd ga 9 


b 


I 


do - Vy 


Va and Vp being round by solving the normal equations. Naturally these 


normal equations become the same as those of exercise 15 if w=1 
throughout, 


Exercise 15. (a) The formula to be fitted in the two previous exercises 
can be taken in the logarithmic form 


log y = log a + bx log 6 


Suppose. now that. y" be. written for log y, a’ for log.a, x" for Iogam 
B for b log e; then 


We now take 


N 
* 


a =~ (a,' + Box) (Y' = log yaa 


wherein aj means 10g a , and Bo means byl log e, The derivatives of f 
are as follows:* 


2 me Stat Oy Ges 
f, = -B, Vi toh dee fy oe ee 0.434/y 
dy'dy 


fare Sly hte 


ee em me a me me ee ee me ee cae me ee ee ere ee ee ee ee re ee ee ee ee ee ee a ee ee 


* It is convenient to remember that log ¢é = 1/in 10 = 0,454 = 1/2.30 cee 
log means base 10, 1n means base e (logarithm naturel). 


Di 
a 
ee 
oe 

















q 
in 
2 
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Suppose that x is free of error, but.that the weight of an 
observed y coordinate is w, which may vary from point to: point, ~ (If 
all y coordinates have equal weight, it is easy to put w equal to 1 in 
what follows). With Wy = ©, the first term in L drops out, leaving 


L or 1/W = 0,4342/y2w 
It will be noticed from the result of exercise 8c, page 30, 
that if w is the weight of y, then y®w/0. 4348 is the weight of y' or 


log y. Suppose that on this account we sect 


w' = y*w/0.4342 = (2.30y)®w (See p. 30) 


then w' is the weight of y', and 


If w (the weight of y) is constant throughout, then w' 


is not, and 
vice versa, (Cf, the remerk appended to exercise 18). 


y 


Table 2 of scction 16 will have headings as follows: 











eee Cm + ee ee 








h, or | fat/VL fp/VL Vw't, 
Point No, = -Vw'! eo! Sum 
The normal equations will be 
No. Vat VB A Cy Co Sum 
I [wt J [xw' J -(w'f,] 1 Q 2 
2 a 2 Meer «a oem 0 A - 
3 batt 0 0 = 








We perceive that these ard similar to the normal cquations of 
Sxerciee J, DUt with w’ in place of w, and f in place of F.~-.More 
precisely, the comparison is this: 


i 


Peo. oe y =, a8) OX, x ivee of error, “yor weight w, 


Here, alr ht tae OG aus secret Wire hira aelaeshyuntnisoesgn 4 unmuple a w!, 
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This means that we may fit the equation y = agbX by writing it in 
the logarithmic form 


log y = log a + bx log e 


and treating it as a linear equation in log y and x, at the same time 
giving ln y a weight just y* times the weight of y, or log y a weight 


e 


(2.30 y)® times the weight of y. 


Remark 1 It is customary among computers to fit the exponential 
equation y = ae Ox by taking logarithms and treating it as linear in log 
y and x, but it is not so usual for them to change the weighting to 
correspond to the logarithms. The neglect of the factor (y 1,434)* not 
only produces somewhat incorrect results for a and b, but also invalidates 
the reciprocal matrix and all calculations made with it on the S. a. of 
a function of the parameters; Moreover, under such circumstances, the 
left-most entry of Eq. III no longer contains g*. In fact, all calcula- 
tions are vitiated, the spurious values for a and b so obtained being. 
about the least objectionable feature. See also some remarks at the 
conclusion of sxercise 18. 


(bd) The left-most entry of Eq. III in the solution of the normal 
equations contains Dwlyops - Yealc)*. The left-most numbers appearing 
in Eqs. 5 and 6 are the weighted sums of squares removed successively 
by adjusting a and then b (see exercise 3 of section 17 and exercises l, 
4, 5, 10-14 of this section). These statements would not be true if one 
were to neglect the factor (2.30 y)*® for the weight of log y. 


Note that-in the normal equations of part (a) it is permissible 
to use a,o'= 0 and Bo = O, in which event 





faum 2) OE 10g sions 


whereupon ths normal equations will be 








No, at B 1 C1 et C, ? sunt 
I [wt J [wtx] [w'y] 2» 0 | = 
Pa Lw'x® j [w'xy | 0 “i | A 
3 . | [wt YY] 0 Osi : 


(wt = 2.30% y*w as on the preceding page) 
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giving a' and B directly. As in exercises 1 and 5, no question of an 
approximate value of B enters for ths calculation of the w', sincs Wx 
is infinite (x free of error), but more decimals are required than when 
good approximate values of a and b are used; see exercise 1d, 


In the solution of these normal equations by the routine sxhibited 
in section 16, the minimized Dwlyops - Yealc)* appears in tho loft-most 
entry of Eq. III, as usual. Directly above it in Zq. 5 comes the reduc- 
tion brought about by changing a from 1 to its final value, and in Eq. 6 
appears the further reduction accomplished by ‘turning the logarithmic 
line from the horizontal through the ehzle are tan B. 


Exercise 16, In fitting the equation 


y = agh% 


with x and y both subject to error, we may take F as in exercise 13, 
whereupon 
box : 
Fo = Y - a6 {(X and Y observed) 


Here we have use for the additional derivative F, = -by, whence 


x 
Lior tee y fw + L/w, 
(If x is free of error, the lst term of 1/7 drops out and leaves 


W=w,, the situation assumed for exercise 14; if y is free of error, 
the 24 term drops out and loaves WW = wi/ry}. 


Since we ars here taking the case whers x and y may both be in 
error, we set up table 1 of section 16 with headings as follows: 








Ae, AU ie = Fy = 
h =-by. 
je ale Wx phy or 1/W or 1A“ -y/a -xy rae 








From this is formed table 2 with headings exactly like those 
of exercise 14 but with w replaced by W. Likewise, the normal equa- 
S10nS Wil) be ‘symbolized as in “exercise 14, w replaced by W. In fact; 
once table 2 is set up, from then on it is immatzrial to the computer 
whether on: or both coordinates are subject to crror--a statement that 
holds zood in any problem of curve fitting. 


The solution of the normal equations will give vg and vp. The 
minimized sum of weighted squares (¢*) will appear in the lcft-most 
entry of wq. III, the portions removed by the successive adjustments of 
a and b falling in Eqs. 5 and 6 directly above @*; cf, exercise 3 of 
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17, the exponential] 


section 17 and exercises 1, 4, 5, 10-15 of this section. Here, ¢* = 


ae) 


Di wyVx% + WyVy¥ ), both x and’y residuals being present. 


Exercise 17, To use the logarithmic form 
log y ='log a + Bx (B = b log é 
4 = 'OyeS4 *'b) 


or y' v= a OF I Bx 


for fitting the exponential y = ae>* when x and y are both subject to 
error, oné would define f as in sxercise 15, whereupon 


te = xi -(ad! + Box) 


The derivatives of f are as in exercise 15. L or 1/W will now 
have two terms, both coordinates being subject to error; in fact 


L or 1/W = Be/w, + 1/wy' 
w.. being the weight of log y. The normal equations will be symbolized 
exactly like those of exercise 15a, but with W in place of w,.'. The 


left-most entry in Eq. III will be the minimized sum of squares ¢*, with 
the remarks at the end of exercise 16 applying here as well. 


The analogy with exercise 4 is perfect throughout. 


Exercise 4 Exercise 17 
y=at bx i orth Gap conte 
L/w = v® /w, + L/wy 1/w = BP fw, + 1/wy! 


All the remarks and notes of exercise 4 apply here if y is replaced by 
yor 10g Vv. a by ‘a by Beng Wy by Wy’. 


It is possible that in some problems, w,,'. might be constant from 
one point to another (in which case w, is not Constant); then if Wy is 
also constant, we have a situation toward which the remark at the end 


of exercise 8 is directed. 
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[Sec, 19) Be. 18 
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| Exercise 18. (a) Take 

| f = log y - (log a + Bx) as in exercise 15 

pie y- ae otf tt 13 


| and suppose that x, y, a, and b take on small increments denoted by 
Sx, otey” Prove that 


oF =. y 6f log e 


| hence at any point, F, = yf, log e to within higher powers of f, of Fy. 


| (b) Thenee prove that the normal equations in exercise 14 for ~ 
| fitting y = as>* will give the same curve, i. e., the same results for 
a and b and for 2wlyopns - VYeaic)®, as the normal equations in exercise 
i 1da for the equivalent logarithmic form (exercise 15), except for dis- 
| crepancics involving the squares and higher powers of residuals, the 

| logarithmic form being slightly more accurate.* (Hint: note that if 
Vga is small, va! = 0,434 Va/a. Also, so far’ asa and b are concornod, 
the top normal equation in exercise 14 may be multiplisd through oy a). 


The same comparison holds betwecn sxercisss 16 and 17; but with 
b ¥ Oo, and x and y both subject to error, there is not so much advantage 
in the logarithmic form, 


Remark It may be worth whils to pause for a thought on the 
factor (2.30 y)* which is required for the propcr weighting of log y 
(see exercise 15). Considcr the following thres points, artificially 
constructed for the purpose:** 








, i tid ann Bae Ses ingot a 

| ar ie ar 10 C 1 

10 ? 2 0.7762 
100 10 2 1 

Av, ‘= "9 Av, = 0), 9261 





x fres of error, all three: values of y- have weight unity. 




















[ 


ea ae a a a a a a a ea a a a a a a ee a a a ew ee 


* My former pupil Mr. X. A. Norton first pointed this out to me, 


**This illustration was ceveloped in some: dorrespondence with Professor 
W. L. Gaines of the University of Illinois, extending between 1932 and 
1938, me ere , 
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Now it is psrhaps evident that in fitting 
y = ae 
or log y = log a + Bx (B = bd log e) 
the result must be B = 0, because there are only three points, the end 


valuss of y being equal. One might therefore be tempted to take the data 
and fit the horizontal line 


y=a (since e° = 1) 
getting a=yr=9Q 
with Yres® = (10-9)& + (7-9)* + (10-9) 


iH 


LOR te Fe 4 ALO”. eB 19F 66 
This result is in fact correct. 

It is interesting to see the same result come from the logarithms, 
correctly weighted, With B = 0, the equation for log a (from the top 
normal equation in the note appended to exercise 15, or simply by weight- 


ing log y proportional to y#) is 


fy | tog a = by? logy} 


That is, (10°. 4,7" 4+ 10°) log a = (10° + 7% x 0.7782 +, 107) 
whence log a = 238.14/249 = 0.95635 
and a= 9,044 


which is a trifle high, but good. The small discrepancy with the exact 
result (a = 9) arises from the size of the residuals in y, which are 
hardly small, though neither huge. Smaller residuals would have given 
closer agreement; larger ones would have produced a greater discrepancy, 
The factor y* has a tendency to over-weight the large values of y when 
the corresponding residual is large, but this overcompensation is 
greatly to be preferred over none-at all, as will -be evident in the 
next paragraph, wherein the weighting factor is ignored, 


If one forgot to weight his logarithms in proportion to y®, he 
should have had 


log a = [log y]/3 = (1 + 0.7782 + 1)/3. = 0.9261 


and a = 8,435 


which is spurious, Leing considerably below the correct value a = 9, 
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The factor y® takes care of the Change in scale that accompanies 
the taking of logarithms. The y values may all have the same weight, 
i but their logarithms do not, No matter in what form the relation be- 
tween x and y be written, the two terms F,F,/w, and Eo /w. in L or 
ifil can be relied upon to perform the same service as (2.3 y)* does 
for the logarithmie scale, 


| This is an illustration of the fact that if the procedure of 
section 16 be followed, it makes no difference how a formula is written; 
one form will give the same curve as another form--except for disturb- 
ances arising from the neglect of 2d and higher powers of ths residuals, 
but these are not usually of much consequence, 








THE EXPONENTIAL WITH A 
LINEAR COMPONENT 


Exercise 19. Given the equation 


y= ago® NRE CG aE | 











to be fitted to n points, Write 





ee ym ae - ox - d 


The derivatives are 


I e Bop 3 2) 
I Weer scbe* fe, FT = 1 


1/W 


it 


(abe>X + c)® /wy, + 1/wy 


(The lst term of 1/W is missing if x is free of error, the 24 if y is 
free of- error), 


|) The formation of tables 1 and 2 of section 16, and the formation 
F of the normal squations and their solution, is by this time a well- 
learned routine process. Here there are four parameters and hence un- 
knowns, Vg, Vp, Vo, Vg. The derivatives Fy, Fp, etc,., also F,, for 
tables 1 and 2, are calculated with the approximate values a,, by, C4, 
and dy, arrived at somehow, as by graphical msthods, or previous experi- 
ence, 
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The left-most entry in Ea. V of the solution of the normal equa- 
tions will give the minimized Riwve WyVy) . The entries just above it 
In hos ley lS ra ana re wi new the reductions in the weighted sum 
of squares arising from ths successive adjustments of a, b, c, and d. 


Note that if only the y coordinates are subject to error, the 
left-most entry in Eq. V will give Dw-res*, the deviations being meas- 
ured in the vertical (i. 2. parallel to the y axis), Moroover, if all 
‘ y coordinates have the sams waight (unity), then W = 1 throughout, and 
tables 1 and 2 of section 16 coalesce, 





Note that with a formula of this kind, there is no possibility 
of making it linear by such a device as taking logarithms, and for this 
reason, it and others like it have beon called insoluble, Fortunately 
the solution is entirely straightforward, 





THE GENERALIZED HYPERBOLA 
AND ITS LOGARITHMIC FORM 


Exercise 20 Given the cquation 
b 
y = ax 


to bs fitted to n points. Here we write 


Fey- ax” 7 
whencs F 
Pe yy ax (a and b being replaced by 
Go: and bo) 
The derivatives of F ere 
ee -by/x, Fy op he 
Fe = -y/a, Py (eucy iim Be 


Lior LAT = neat etm + 1/wy 


The headings for table 1 of section 16 in the general case would be 


h xe cby/fe owe wy, LorifW vi or 1MW 
(cont 'd,) Fo = -y/a Peay crx pe 








ws 
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It is easy to make the necessary modifications for special cases. 
Thus, if x is free of error, then W = wy and the Fy, and w, columns are 


superfluous; if further, all y coordinates have equal weizht (unity) 
then W = 1 for all points, and tables 1 and 2 will coalesce, On the 
other hand, if y is free of error, then 1/¥ = by /a®wy and the w 
column is omitted. From table 1 is formed table 2 with headings 





h VF, or Fo/VL VWEp or Fp WWE ote eA. Sum 


- 











(= ee rm on 











a ee ee ee 


The usual sums of squares and cross-multiplications from table 
2 give the normal equations 











No. ve ot es : oe Yee Sum 
I (Wai, J (WE Fy) -[WFFo] 1 0 - 
2 [WFLF DI -(WELF oJ O Q 
S (WFoFo] Ol ph a6 - 


A NS A A RN 





Exeycige\21, ‘The equation y= ax® of thé preceding exsrcise may be 
turned into the logarithmic form 


Oey = lowe a + Loe x 


or yi era tear hx? (as in "Exot 


fei ye late + bx! } 


Ge. ashusual 


The derivatives of f are 


f, = - 0,434 b/x f, = 0.434/y 
ei Tate Tp Foe! 
Then L or 1/W = O.4842 (fx w+ 1/¥ wy) 


iH 


oF /w,," + 1/wy! 
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wherein 1/w,! = 0,4548/x¥w, = ‘}/wt, of x' or log x (see exercise 
Sc. Gne 
L/w! similarly defined. page 30) 


The headings for table 1 of section 16 will be 
| a Be ae #8 datas ; 
Point No, Wy Wy! Wy Wy" L/w VL te ty ye 








(See the remarks under table 1 of the preceding exercise, fg is not 
listed, being constant). 


The headings of table 2 will be the usual ones, as in table 2 of 
the procveding exercises with £ in placesor f., 


The normal equations will be symbolized precisely like those of 
Set 1 in exercise 44 In fact all the remarks and notes of exercise 4 
can be translated directly to the present problem. The reason is ob- 
vious: we have here a line in the variables x' and y', with weights 
w,' and Wy! The two terms of L or 1/il seen above take care of the 
change in ths form of the function from exponential to logarithmic, 
In fact, we could say that 


L/ Ms tent a dy at Eitysi te} = fy fy/wy + fyfy/wy 


as in exercise llb page 30. 


Exercise 22, Prove that the normal equations of exercise 20 for fitting 
y = axO will give the same curve, i. e. the same results for a and b 

and hence for ¢“, as the normal equations of exercise 21 for the equiva- 
lent logarithmic form, log y = log a + b log x, except for discrepancies 
involving the squares and higher powers of residuals, the logarithmic 
form (cxercise 21) being slightly more accurate, especially if x is free 
of error, (Refer back to exercise 18). 
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THE HYPERBOLA WITH A 
LINEAR COMPONENT 


Exercise 23, Given the equation 
y = ax> + ¢ + dx 
to be fitted to n points, Write 
foe y = ax? eee dx 
F, at any point is found, as usual, by giving x and y their observed 
values at that point, anda, b, c, d thoir approximate valucs ae bay 


Co, dg (found somehow). 


The derivatives of F are 


acaba -d) | Fy, =, 
Foe ete = “ax ln x, Fo=. <1) 
ays =: 


whence 
L or 1/W = (abx7* + aan. * 1/wy 
as doe wy if x is free of error) 


Tables 1 and £ of section 16.are made up, and the normal equa-. 
tions formed and solved by ths usual routine. Eq. V in the "1" column 


will give the minimizod value of. 6* or U(w,V¥% + wyV¥), the successive 
reductions in the weighted sum of squares appearing in Hqs. 12, 13, 14, 
and 15, as usual (see exercises 1, 4, 5, 10-16). 


The reader should refer back to the notes eee 3d to exkoroles 
19, which ‘apply here as well. 
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the hyperbola] 


Exercise 24, Given the equation 
u= ax + by° + dz 
u, X, y, and z possibly all being observed.(This equation is used by 
Professor W. L, Gaines at the University of Illinois in his work on 
nutrition and lactation). Take 
F=u - (ax + by® + dz) 
F. as usual 


e) 


The derivatives of F are 


a ie at c-1 so 
By ein) PEs ae, Aes Dy micah Benes 
Fa = -x, Fp =-y®, Fo = -by° ny, Fg = -2 
pS ee! Come hal eyelet 
it) Wy Wx Wy Wo 


Here we have a problem in four dimensions; 1/W contains four terms, 
The first term is absent if u is free of error, the second 1f x 18 Dee 
OL ierror, etc. 


The headings for table 1 in section 16 would be as follows: 





ae 





h, or 
: el a 
Point No, Wie Wy, by By Wy We L/W 
Cont 0. 
NOR. eee ye Fa 
Lil a. C F 
i wee = -ye -by° in y = 9g ce) 





ee ee 


Some of these headings will be omitted if any of the u, x, y, or z 
values are free of error throughout, 


For table 2, for by obvious divisions from table 1: 


a ne re rr 


VF g VE 6 





h 
hee VWF,  VIFp Sum 


Point No, 


The normal equations will be 
routine manner, Ea. V will contain 


D(wyVyF + wyV3& + wyVyP + woVQ*) in 


ee 


formed and solved in the usual 
ths minimized value of 3* or 


the "1" column, the successive 
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reductions in the weighted sum of Squares appecring in Eqs. 12, 13, 14, 
and 15, as usual; see exercises 1, 4, 5, LO LG. 23) 


If only the u coordinates are subject to error, 4% Se 


“oreover, if all the u coordinates have ths same weight (unity), then 
W = 1 throughout, end tables 1 and 2 coalesce, 


The last note appended to exerciss 19 applies here, 





MISCELLANEOUS 


Exercise 25. (a) Given the equetion 


a 


ROT eae hy aed 
to be fitted to n observed points, With 
F = u -(ax + by + ez) 


show that 


SS aS ES SS A eI a AT 
Rare ere 


Lf = lf, + a /w, + OF fity + fit, 


and that the normal equations are 





: No. 7 Vy Vo = 1 Ge Oe Oil sium 

| I Cwxx] {Wxy] [wxz J -[WxF 9] f 0 0 - 

: 2 CWwyy] (Wwyz J -[LiyFo ] ) 1 0 - 
3 (W2z] -[W2Fo] 0 0 1 - 


) 
) 
| 
| 
| 
} 


4 (WF OF, J o) 0 0 - 





(b) If it is desired to solve for a, b, ¢ directly, without the 
use of the -pproximations 29, bo, Co, the unknowns in the normal cqua- 
tions would bs a, b, and c, and the "1" column would bs 

Cwxu] 
[Wyn] 
[Wau] 
(wuu] 


be extra decimals being required for accuracy, as mentioned on page 119. 
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(c) Prove that the minimized sum of weighted squares is 
po? = (Wuu] - [WxulJa - [Wyul]b - [WzuJe 


(Sce exercise 3a in section 17) 





Remark. If u alone is subject to error, and of uniform precision | 
(unit weight) throughout, the only change in the normal equations would 
be that W would not appear, being unity throughout. The minimized sum — 
of squares would be 


= {uu] - [xula - [yu]b - [zuJe 

(This equation is used a good deal in some kinds of statistical work; 
see e, g. page 160 of the 6th edition of Fisher's Statistical Methods 

for Research Workers, on which the above equation appears as i: 

Sly-¥)* = S(y?) - bs Slxiy) - be Slxgy) - bz S(xay) 


this being the sum of squares after fitting 


Ye = bit. eth Date 


See example 3 in section 20 for an illustration. 
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20 . Three ek amples in curve f fitting. 


Example 1, fitting an isotherm, Data taken by Michels et al., Proc. 
Royal Society (London) 153A, 201-224, 1935. The equation to be fitted is 


(y denotes pv, pressure times 


Viele ot 0k 2 tres dx 
volume; x denotes density) 


The parameters are not independent but are subject to the condition that 


y =1lwhen x=l 
Then Qu= 1. = b= ¢ = d 
and y = 1 + (x-1)b.+ (x* -L)e + (x* -1}4 
Weights: All y coordinates to have equal weight; x free of error. 
Let F=y - {1 + (x-l)b + (x®-1)e + (x4-1)a} 
Derivatives: Peak ok we ure esd ea at amex Apo 
Fy = bs ..» Fycis not needed: since x is free of error. 
W= 1 at every point. 
The following approximate values are known from previous experience: 
bo = - 0,006837046 
Co = O,00001 1392 
dae yO A514 
Let y' = 1 - ,006837046 (x-1) + .000011392(x*-1) 


#20, LOlal eet) 


Then Ee yene - ¥! 
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Tables 1 and 2* (formed from the original data). 











Point Fy Fo = ORES or Sr ae 
1 1, 77x10 3.51x104 1.24x105 0.552107 * 7 OF 
2 2.25 5.49 3.03 0.79. 11.56 
3 mS re 7.93 6.30 2.07 19.02 
4 3.18 10.74 11.55 3.09 28.56 
5 3.66 14.15 20.04 6.22 44,07 
6 4.14 Loe 32.39 10.05 64.57 
i, 4.61 22’, 20 49.31 14,84 90.96 
Sum £2.33 82.01 123.86 37.61 265.81/ 


The normal equations are formed in the usual manner by cumulating 
squares and cross-multiplications from the above table. The powers of 
10 in the headings bring uniformity in the denominations of the columns. 
The Sum column provides a check, which should never be omitted; it is 
formed regardless of the powers of 10; in fact no attention is paid to 
the powers of 10 until the end, when the solution is decoded. Why is it 
better to start off with 100 rather than 1 in the C,, C,, and Cz columns 
of Eqs, I, 2, and 3, for the calculation of the reciprocal matrix? 
Perhaps 1000 would have been better than 100, 

Note the symmetry in the reciprocal matrix, which is found between 


the vertical lines in Eqs. 11, 12, and 13, 


me me me ee me me eee eee ee we re me ee ee ee ee a a ee ee 


* [In this example, W = 1 throughout, with the result that tables 1 and 2 - 
mentioned in section 16 are identical. The minus signs in the headings 
avoid minus signs in the table. 


Reh at 














a 








Pe 98.03 x 1078 + ie ’x 








28.99 x 1078 =. 8 '> VONoe S074 


remember that a is a func- 
See also exercise 


es vs ~155= [Sec. 20, 
example 1] 
Normal ‘equations: (the solution follows the form outlincd in seetion 19). 
aaa . ‘ie 
atOW lv, 10§vq = i c oc; ep Sum 
we 7 253 302.94 497,86 (151,.06-107-*] 100 0 @) 1129 ,.38,/ 
a ve 1256.98). 2155.67 (654.02 an?) 100 @) 4449 ,61/ 
. 4066.38 1233.79. »0 0 100 8053. 70/ 
fay 4.66 0 0 0 2413. 55/ 
Factors ee 
-3,907377 -1183.68 -1945.32 -590.23 =S90 72.010 0 -~4412,.90 
53.30 £10, 35 63.79 -390.74 100. QO 36, 70/ 
=| i oa? ieecnmaneentncynceee ees ; 
| -6 .421556 -3197 02 -970.01 -648 16-860 0 -7252,36 
| -3,946701 -830,20 -251.75 paver 304 671 6 -144,85 
| 39.16 Le. 03 | 899,97 -394.67 100. 656.497 
: -| * at 
| -1.948379 =o 3) * 1-194,84 0 O° We enao te 
~1,196797 ~76 34 | 467.635 -119.68 0 -43 93 
-0,3071320 -3.69: ~27641.-"“bel 8-30. 7 ee eOies 
233 -3,61 1.54 -30.71) oe ser 
| Merefore d(w¥") = ,33x107*; o®(ext) = .33x107*/4 = ,08ex10 
“13° 10vy = 036136 236.75 =9e°0o 1.22.96 
eL2 LOoe a = , -,015361 06708 41.65 -10.08 
Pll LO*y a .3071320 SRG -£0708 2,55 16.76/ 
Dy eae xlO~* b = -.00683705 - .0°36 = ~,00683741 
a POvexlO 7 Che MOS) 1392) 290) OF 150 27 41K 07 
= + .307x10~° da 0? PS]. mer OF 207) me OF 156 
| q Pease ot ab baie s ce =e ae 10068 R600 
| :, 
| i S.E Bot o> 2eegeoen, 10>” "-gatext) = 19,3:x 10. 
ee tee GLO OR fexty= 3,40 x ee 
" io a sev, eee sy: 1072 72% 2 (ext) = 0.209x 10729 
" " a = o8(ext) (236.75 x 107* + 41.65 x 107° + 2.55 x 10722 
| 
| 






results are: 


17.7 ioe 207”) \i@seaind. -10r: 
Lion Or boece- and d, 
1 following). 
a = 1,0068260 + .vo00042 ° 
b = -.0068374 + .0000044 | S.Es 
Gee rth) te O18 | 4 
d = -.0°156 + .09046 | 


. estimated from 
degrees of freedom, 
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Exercise 1, Show that when corrected for powers of 10 the reciprocal 
matrix is _ 


236.75 x 107° ~98,03 x 1075 22.98 x 1078 


Roa? = “98.03 x 107" 41.65 x°10r° -10,08 x 107° 


Be. 98 a0.” ~10_067x 10"? 2, OG) x! 0 wae 


These are the figures that were used in writing down the S. Es. of a, Db, 


ec, and ad. Evaluated as.a determinant, this gives A™* = 4.6 x 1077*. 


‘Exercise 2. The evaluation of the determinant of the coefficients is © 
A. = (77.53 x 107)(58,30 x,10*)(39.16 x 104°) = O Teepe 
(See Ex, 1 of section 17, page 108). This result is not exact; the discrep- 


ancy arises from instability, and could be overcome by carrying more decimals, 


Exercise 3. Prove the S.E. of the curve at x = 1 is 0, and that at x = 0 
it is.the same.as.the S.E. of a. .Why is the S.E..of the y intercept & 


practically equal to the S.E. of b? Argue geometrically. 


As often happens in curve fitting, these normal equations are un- 
stable, The trouble comes from the fact that there is not enough spread 
in the abscissas to determine accurately the coefficient d. One of the 
most sensitive tests for instability is to compare the direct solution 
(already found in the "1" column of Eqs. 11, 12, and 13) with that given 
by using the reciprocal matrix as a multipliers by such means we get the 
reciprocal solution (pp. 111-112) 


10vy, = {(151.06) (236.75) -.(654.02)(98,03) + (1233.8) (22.98)} 10°27 


0.0237°107* 


it 


and in like manner (which the student should undertake .as an exercise), 


10°v, = -0,0508°107*,  10°v, = 0.2500°1074 


(Sea, 220, 
example 1] 


These are in disagreement with the direct solution found in Eas. 
11, 12, and 13, and thus instability is indicated. The direct solution 
satisfies the normal equations to the last decimal, but when there is 
See hey many other solutions not too far away could do the same 
thing .* The reciprocal solution, however, does not satisfy the normal 
equations, the actual numbers being 111 eeeinet 151.06, 483 against 654.02, 
919 against 1233.79. 


The insidious thing about instability is that its presence may go 
undetected, For instance, if here we had only the "reciprocal solution", 
and had not tried to check it by substitution, we might have accepted it. 
The use of the reciprocal matrix as a multiplier is in theory very 
fascinating, but as a practical matter in curve fitting we should not’ 
lose sight of the fact that it occasionally does more harm than good, 
Fortunately it does work to good advantage in many problems, as seen 
for instance in Ch. 5 of R. A. Fisher's Statistical Methods for Research 
Workers. In section 12d, also, the equations were stable and no diffi- 
culties arose, It is interesting to see what would be the sum of squares 
if the term dx* had been dropped. From Eq. III we find [cc.2] = 


39.16 x 10°; this multiplied by do® or (0.0°1514)% gives 0,09x1078, 
Which added to 0.33 gives 0.42 x 10~® for the sum of the (ygpg - Yoaie!® 





that would have resulted from fitting the curve y = a + bx + cx®, We 
then find 


o#(ext) = 0.42 x 1078/5 = 0,084 x 10-8 (dx omitted) 
and we already had 
o*(ext) = 0.082 x 1078 (dx* included) 


50, as we might expect from the instability of the equations, the indi- 
cations are that the data are insufficient to determine d very accurately, 
in spite of the fact that the value of d turns out to be triple its 
estimated S.E,. Other solutions could be found for the normal equations, 
satisfying them to the last decimal, but allowing d to be considerably 
different from that which we happened to find here,’ 


ee ee me ee mm mm ee te ee te ee a em re mee mm mem ee mee ae ee me me ee ee ee eee me cee ee ee we ae ee ee a ee ee ee ee ee ee ee ee 


* See the author's paper in Science, 7th May 1937, cited in section 17. 
Further work by L. B, Tuckerman of the National Bureau of Standards is 
in progress and may appear in Science in 1938 or 1939, 
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. Example 2 (illustration of Ex. 10 of section 19), The polynomial 


Mero ar eins a5 0 eas 


fitted to the following points in the x, y plane, x and y both subject to 
error, 


Observations 


X and Y denote the averages of N readings. s = S.D. of N readings 





Se EE | nT SS AR EEE 


On the y coordinates 






| | 

Point No. | 
i 

i 











| X(cm.} N s db eee N a 
ia Dawe 1 Clg ek eee Satie aie 0,306 0,087 10 .00258 


The table shows, for instance, that at point No, 1, there were 5 observa- 
tions on the x coordinate, their averags being X = -1.92, and their 
S20 292 Oco0e.s 


The question arises how to weight the various values of X and Y, 
For one thing, the weight of any coordinate will be proportional to N, 
but that is not enough; the precision of single observations may vary 
from point to point, and it is evidently different for the y coordinates 
than for the x coordinates, judging from the s* columns, 
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Previous experience in this line of work, let us suppose, has 
indicated that the S.E. of single observations on x is about 0.5 
em., and on y about 0.05 lb. In order to investizate the precisions 
Lor ae particular set of observations, we make 2 graph of the values 
of s*N/(N-1) for x, and of the same inane for y, both plotted against 
x (y would do as woe. s®N/(N-1) for x (or y) at any point is an 
estimate of the square of the S. &. of the single observations on x 
(ory) at that point. 

oO 
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Fig. 12. Estimating the precision of the observation. 


Although there is fluctuction of the estimates, there is no trend; 
moreover, the weighted average on the x plot is not far from 0.25, and 
the average on the y plot is not far from 0.0025, so it seems reasonable 
to conclude that the prior values of precision should not be changed, 
Accordingly we take 0.5 for the 5. EH, of single observations on x, and 
6,05 for the S. H. of single observations on y, uver the entire range. 
By recalling that weights are inversely proportional to the squares of 
variances, we see that a single observation on a y coordinate has 100 
times the weight of a single observation on an x coordinate (v. Eq. 16 
p. 13). As a matter of convenience, then, we take unity for the weight 
of a single observation on x, and 100 for the weight of a single one on 
y. This is equivalent to setting o = 0.5 for observations of unit 
weight. The values of w, (next page) are then the same as the numbers 
N on the preceding page referring to x, and the values of Wy are 100 
times the numbers N referring to y. (If the precision, of single ob- 
servations on either x or y coordinates were variable over the range 
of the points, obvious modifications could be made in the weighting). 
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For the function being fitted we take 


F=y- (at bx + cx*) 
ee Px Sime Py nC (See exercise 10 
Pe staes case see aia -x* of section 19) 
and L/W = (b+2ex)*® /w, + ify 


By passing the curve y = a + bx + cx® through points numbered 1, 6, and 
ll, the approximate values 


a, =)0,1855 
Do = 0.0526 
Co = 0,00069 


are found (see note on the "method of selected points" page 34). This 
selection makes F, = 0 at points 1, 6, and 11 in table 10s 


We are now ready to fill in table 1, the 3d step of section 16. 
1/W is computed by using the value of (b+2cx)® in the 24 column (see the 
formula for 1/W written out above). 




















Table L. 
oa x * 
nae (b+2cx)* ca ies 2M Li oe -Fy -F, re 
1 0.00253 5 TOOG IO, OOLSL GLOSS i -1,.92 SLO¢ 0 
2 264 6 900 sits) 39 1 =) LS 1.32 +07Gge 
3 281 a 800 165 Al it 0,04 0 + ,O3F 
4 294 8 700 180 42 a4 0.86 0.74 +e 
5 412 9 600 201 45 L 2.0L 4.04 - ,027 
6 oe” 10 500 233 i Revs 8.94 0 
” 338 9 600 204 45 uu 3.69 13,62) +e 
8 363 5 ' 1000 175 4B 1° 5.20 27.04 =ama 
9 o79 7 800 179 42 x 6.09 37,09 ‘ies 
10 398 6 900 eT 42 as Tee 51.55 = eae 
11 409 5 1000 182 43 ui 7.80 60.84 0 
12 432 6 900 183 43 fi 9.05 82.45 + Gee 
eee are 


Since - Fg = 1 all the way down, it would ordinarily not be listed, 
** Computed with ao = 0.1855, by = 0.0526, c, = 0.00069, Why is Fy = 0 
AC DOINtBONG, Wyo, on ue ; 
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The 4th step of section 16 is to form 
Table 2 (The matrix*) 
Point No, VWF, VW LE AW Be Vil Fo Sum 
1 £0.8 -4.9 x 10 OPO ey aL Oe O e120 
2 25.4 2.90" Osan 3.8 x 107% 26.6 
3 24.6 eal 0 cele a 33,8 
4 20.6 ee) s O28 " en " PA ey, 
o oawo 4.5 3 oo " =-5 0 Mi ren AY 
6 20 : Gre " Lita! “i @) 28.7 
7 al. Be S 3.0 bi 10.2 y 43.5 
8 24.0 ec " 6.9 Ly ~4.3 Se 33.7 
9 eoen 14,4 4 8.8 Ay 4,0 "i 5053 
10 Be yaa Afra 8) H Ware nN es! he Md 48.5 
ci BO. Loe e Lars " O SLeiaas 
le 25.4 Be " ds u Ties # 6575 
Sum 282 .8 mpewte 68.2 14.9 462 .5/ 


Note that the minus signs at the headings obviate the writing of minus 
Signs all the way down. Also note that the Sum" column is formed with- 
out regard to the powers of 10, 


The next step is the formation of the normal equations by the 
usual cumulation of squares and cross-products from table 2, The solu- 
tion is carried out by the routine outlined on page 107 and used previously 
On pages 64 and 65, and in the preceding example. 


me me ee ee ee ee a ae es we ee 


* So-called because from it is formed the normal equations. Moreover, 
in matrix notation, the formation of the normal squations is the same 
as taking the product M'M, M being the matrix of table 2 of section 16, 
and M' its "transpose", 
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Normal Equations 





No. aM -LlOv,, =a00v 7 
Ai 6686 2250 1601 

2 Logg Liz. 

He 859 

4. 

Factors 

5 733353 -744 -534 
ge 855 E37 

6 - .25946 -383 

ti - 68655 ~403 

IGN 73 

8 - ,053395 

9 eLeOay 

40 - ,02739° 
IV 
1® ‘Subst. 11 & 12 into I ae 
Poe Supst. LE into a) ~LOVs, 
Pee One Go) ent o el EDA 


From Eq. IV, 2* 


From Eqs. 13, 12, and 11 we 


Va 
V5 


a 


cxample 2] 








The adjusted values of the parameters are therefore 


ee ee a ee ee eee ee we ee ee ee ee ee ee ee ee nee ee ee ee 


The factor 107~* holds all the 


** The decimals in the C,, ©,, 
ignored in the 


a 


Sum column, 


hosting Adi Ce Ge Cr Sum** 
3579x1072 100 0 0 1097 4/ 
16 @) 100 0 5066/ 
16 ) 0 100 3697/ 
296 0 -O 0 684/ 
119 ~33,35 0 0 -3660 
-103 — -35,35 100 0 1406/ 
- 85 ~23.95 0 @ - 2628 
fo 22.90 -68,.66 0 - 965 
a - 1.05, -68266, “100 104/ 
mS ats 0 0 - 586 
See Buen 12 0 169 
0 @) 2 -% 3 
265 —"9 14 -3 270/ 
= .09330| .02812 -.02915 -.01433 
=~-,13930] -.02915 7627 8 -,9406 
= ,02740/-.01438  -.9406 1.370 
= 2,65 
see that 
= ~0,0093% 
=O, 00139 
= -~0,00003 
= 0,1855 + 0.0093 = 0.1948 
=Q.0826 - 00014 400.0512 


= 0,00069 + 0,00003 = 0,0007 


ee ee ee ey ee ee ee ee ee ee ee ee ee ee ee ee ee ee 


way down the myn column, | 


ee ee ee et es ee ee ges ee 


and C, columns were added later, and are 
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The matrix reciprocal to the coefficients in the normal equations 
is contained in Eqs. 11, 12, and 13 under C,, Cz, and Cg; the powers of 
10 must be adjusted, however, before it will give the variance and product 
variance coefficients for a, b, and, Correcting for powers of 10 
we have 


Te ee 
gan -,0*29 itv ae «OL 55 
Wide. om, 0°95 ,0514 


for the variance and product variance coefficients of a, b, and c. The 
S.Es. of the parameters are then as follows (o = 0.5): 


(S.E. of a)® = 0.00028 o%, or S.E. of a = 0.0084 
(S.E. of b)® = 0.000077 o2, or S.E, of b = 0,0044 


(S.E, of c)® = 0,0000014 o2, or S.E, of e = 0.00059 


The estimated parameters are then written’ 


a Leas ee US: 


S.Es. based:on a 


b= 0.05124 070044 ; 
prior value of a 


4- 


0,0007 + 0.0006 


n®) 
i 


On the basis of standard errors, it looks as if the value of ¢ 
is barely significant. We may wonder if a straight line would fit about 
as well.’ If ‘one were to cut off the term cx® , the resulting increment 
in g* would be roughly [ec.2]c® , as we know from exercise 3 in sec- 
tion 17. The qualifying adjective roughly is needed here’ becauss the 
weighting factors W in table 1 contain c, and were calculated, not on 
the basis of ¢ = 0, but with ¢ = 0.00069. Seeing that [cc.2] in es Fes aD fl 
iS eoweO-* we accordingly add 


11.0400, OOO7& = O57 
to 2.65 and obtain 3.02’ for a rough value of what ¢@ would have been if 


we had fitted the line y = a+ bx, The external estimate of, o would 
have been about 


o®(ext) = 3.02/(12-2) = 0.30 (roughly) 


whereas, for the fitted parabola y = a + bx + ox* it is 


ul 


o®(ext) = 2.65/(12-3) = 0,29 - 
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The. comparison of the two external estimates thus also suggests 
that so far as this set of data is concerned, the line is about as good 
a fit as the parabola. In other words, the term cx* is incapable of 
reducing the sum of squares by as much as it ought to if it were really 
needed, This situation causes some instability, as in example 1. The 


reciprocal. solution is in fact -v, = ~0,0934°107-4, -lOvy, = -0,1325°10 7 


-100v,g = 0.0175°107+, but we shall not stop to discuss it. 


It will be recalled that here we were in the fortunate situation 
of having a reliable value of o, both from previous experience and from 
the "internal consistency" of the observations.* The internal estimate 
was o% (int) = about 0.25, this being the average ordinate in Fig. le 
on the basis that single observations on an x coordinate have weight 
unity, and single observations ona ycoordinate have weight 10, So with 
o*({ext) = 0.29, the internal and external estimates are in good agree- 
ment, neither the function nor the data are laid open to any suspicion 
by a comparison of the two estimates of o. 


It is important to keep in mind the fact that when neither a prior 
value of o nor an estimate by external consistency is at hand, o(ext) is 
no help in judging the fit of the curve; one is bound to come out with 
P(x) equal to about 0.5 if he computes x* with o replaced by o(ext) 
whether the fit is good or bad, as already noted on page 19 ** Neverthe- 
less, two external estimates are useful by themselves for comparing two 
different formulas fitted to the same data--for instance, in deciding 
whether the term cx” is really needed (supra). 


In this connection I should like to quote two paragraphs from a 
manuscript.on. the melting point of rhodium, written by my friends Hew 
Wensel and L, B, Tuckerman of the National Bureau of Standards. 


"I+ should be emphasized that although, lacking other knowledge, 
Gauss's criterion+t of closeness of fit’is the best criterion we have, 
it is far from a certain criterion, It is based upon the assumption . 
that the deviations of the observed points from both of the empirical 
curves are random samples drawn from a normal universe of errors. | 
Unless the distribution of the residuals is a highly probable distribu- 
tion for such a sample no confidence whatever can be placed even in| 
this criterion, 


eee ee ee ee we ee we me we we wee ee we ee ae ee ee we ee ee ee ee ae ae ae a a ee a a ee ee tee ew er eee ee eee ere 


See reference to Birge in section 6c, A comparison of the internal 
and external estimates of o is an application of the “analysis of 
variance", though historically it is interesting to note that physicists 
were using the notion long before the term "analysis of variance” was 
coined, note for instance the reference to Palmer on page 18. 


** Possibly some remarks of mine in the Physical Review 49, 243-247, 
1936, might be helpful. 


+ They refer to o(ext). 
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"As a Simple illustration, two non-intersecting parabolas indi- 
cating a putative discontinuity can be fitted to a subset of 8 and a 
subset of 6 values of the continuous function y = sin x chosen at equally 
spaced values of x, which on the basis of Gauss's criterion fit the 
points better than any single parabola fitted to all the 14 points of 
the set, The critcrion, however, is in this case wholly invalid since 
the deviations of a sine curve from a parabola are not random but are 
systematic." (To appear in the J. Optical Society, 1938). 


Now to adjust the observations, i.e. to estimate where the point 
is that was observed to be at X, Y, we note that there will be a A at 
every point, given by Eqs. 87, page 86: 

Na = WilFo - Fava - Fav, - Feve) 
(The superscripts 
Ag = We(F5 - Pave ~ FoVp - Fove) on F refer to the 
point number) 


We use v, = -0.00958, vi, = 0.0014, Ve = -0.00003, and the F values from 


‘table 1, page 160, We shall not adjust all the observations, but only 
those at points No, 6, 7, and 8 for illustration. Using equations like 
the ones just written, we find that 


1 


0.00n36 '° Sy VOmod te. 99 8 0 OOL A ts 8 G7 OU) 


Ae = 


u 


- 2,544 


he = ——=—— (0,046 - 0.00938 + 3.69 + 0,0014 - 15.6 - 000003) 


i 
ra) 
Lay] 
ON 
0 


1 


Me = OLO0L7S 


(-0,018-- 0.00938 + 5.20. + 0.0014 - 27.0 - 0.00003) 


i] 


LOL) 


whence by applying Eq. 89, p. 91, the residuals can be computed at once. 


t 
rr 
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we 
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At No. 6; V, = (1/w,)A.F,, = 


Vv 


= -£,344/500 = -0, 
A: (1/wy)AcFy 344/500 0,0047 
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AGONG 727 OV 


ti 


(1/wy)AeFy = (22.52) (-.05647)/9 = -0.140 


= (1/wy)AyFy = 22.32/600 = 0,0372 


ATNS Sree V. 


" 


(1/wWy)AeF, = (-10.91)(-.0586)/5 = 0,128 


Hots (1/wy)AeFy = -10.91/1000 = -0,.0109 


These residuals measured off from the observed points in the proper 
direction (see Fiz. 9, page 83) give the calculated points, which are 
supposed to lie on the calculated curve. Actually there are small dis- 
crepancies; the points so calculated do ‘not fall exactly on the curve, 
as is barely evident in Fig. 13, But such discrepancies are trifling, 
being of second order from the neglect of squares and higher powers or 
the residuals. One may simply manipulate the end decimal of V, or Vy or 
both, in order to place the calculated point 

X Pe ae 
(Eq. 90, Dea 
VERGE 


I 


7 y 

exactly on the curve. A precisely similar situation arises in problems 

of surveying, wherein for exact satisfaction of the geometricai conditions 
after adjustment, one often needs to manipulate the end decimal of one or 
more angles and sides (see page 68, middle). 


The adjustment of the observations is important, and now that 
methods are available in the general case (Eqa. 89, page 91), it should 
always be carried out so that the residuals can be inspected before any 
conclusion is based on the summation of w,V|* + w V £. As a matter of 
fact, in this particular problem, a plot of all le observed and calcu- 
lated points directs no suspicion toward the randomness of the residuals, 
but this is always a matter of human judgment. If the term ex* is omit- 
ted, and a straight line fitted to the points, the graph of the residuals 
so obtained gives a faint suggestion that the parabolic term ex® should 
be included, In this particular line of investigation, previous experi- 
ence has demanded the term cx*, which is something this small set of data 
by itself could not settle. 


~167- 
{[Sec. 20, 
example 2] 


Exercise, Compute the x and y residuals for the other 9 points, and 
plot them, 


The S.E. of any function f(a, b, c) is 


one af se. 
Of = 02{0.00028(z)* + 0,000077(a,)% + 0.0000014(~) 


aie ai: iad (ean 
- 2(0,000029)5> ap - 2(0,0000014)5— 35 


- 2(0,0000095)—=- — (seeita. 47.) pare) 


In particular, the calculated y with its S.E. could be written 
y = 0.1948 + 0,0512 x + 0.0007 x2 + o{0,00028 + 0.000077 x® 
+ 0,0000014 x* - 0.000058 x - 0.0000028 x2 - 0.000019 x3}2 
(The terms 0.000077 x® and -0.0000028 x* would of course be combined). 


If the factor o in front of the brace be replaced by 1.96 o, 
the double sign gives a 95% "confidence" band (Neyman's torminology*), 
which interpreted means that at any abscissa, only 1 in 20 of the bands 
so calculated will fail to cover the true value of y, the assumption 
being that the observations actually are normally distributed with 
variances o?/w, and o#/w, about certain "true values", The 95% confi- 
dence band for this particular set of observations is shown dashed in 
Hig. 13, 


ee ie tee eed ed 


* See pages 115 and 116, also J, Neyman, Washington Lectures, (cited 
on page 18), especially the conference on confidence intervals. 
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Example 3, a formula useful in forestry*, 





This example serves the purpose of illustrating three features: 
(1°) the fitting can be done with logarithms, the constant or nearly 
constant characteristic being suppressed or nearly suppressed to cut 
down the number of figures required; (2°) W is constant throughout ; 
(3°) the prior value of o can be expressed in terms of some of the 
parameters, so that, finally, the minimized ¢* can be transformed into 
x*, and the fit of the formula judged on this criterion. All three 
features owe their existence both to the form of the fitted function 
and to the experimental material and procedure. One or more of them, 
however, is apt to be encountered in other work. 


2 
x = the volume of a tree in board feet; 
y = the mérchantable height of the tree; 
Z = its diameter at eee height ; 


then experience has shown that the equation 
Se“.8y 2% 


expresses satisfactorily** the relation between x, y, and z. 


The particular set of data for consideration in the present problem 
consists of 66 points -- measurements on the volume, merchantable height, 
and diameter, of 66 trees. It will not be necessary to display the 
full set of points for the discussion intended here; the first six and 
the last will be sufficient. They come in no particular order of size, 
The logarithms are written in the three right-hand columns, for convenient 
inspection, since thcy will be needed in the fitting. 


Looking at the logarithms (top of the next page) we perceive 
that the first part of the figures, the "charactsristics”, do not vary 
much; in fact, in the Y' and Z' columns the characteristic is unity 
all the way down. What we need to do is to write the formula in such 
@ way thet the variable part of the logarithms is brought into promin- 
ence. This can be accomplished by writing the formula as 


am ee es ee a a ee ee ee a aa ae a a a se ee weer er werner rere ee 


* Iam indebted to my friend Mr. Jesse H. Buell of the Forest Service, 
Asheville, for bringing this problem to me, 


** Francis X. Schumacher and F, dos S, Hall, J, Agric. Res. 47, 719- 
734, 1933; also Donald Bruce and Francis X, Schumacher, Forest Mensura- 
tion (McGraw-Hill, 1935) Art. 140. 
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OBSERVATIONS LOGARITHMS 




















Height Diam. 
x be Z 
(feet) (inches) 





log X Log. ¥. log Z 





1 1.778 1.398 1.140 
a 1.778 1.380 1.146 
3 AOA: 1,462 1,258 
4 2.431 1,580 1,322 
fs) 2.505 1.568 1.334 
6 2.114 1 ag? 1,218 
66 320 54 18.8 2.505 1.732 1.274 





Sums 152.136 102,451 84,090 
x' = at + by! + @2! (x? efits xazetog) 
thereupon lowering the characteristics of x', y', and z* by the harmless 
device of subtracting and adding unity to cach logarithm, arriving finally 
at the form 
xt = gq" + D 8 + CZ t 


where the double primes denote suppressed logarithms, namely, 


KUO Sex! Seok. ey lar. cena) 


y" = ere) Doe yi ned 
2" eug! weleen opi 2 wal 
a" seg! ade pet Gate log a0 =jiba+ PO ae 


By transposing the formula all to one side, we have the acceptable 
form 


sites Gale (a" + by" ae ez") 


The derivatives of f are as follows: 
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Then 


L= 1/N = fyityr/iy, + tytyr/wyr + fyifze/m,, 


x 1/Wy * be fw m ck fw, 


l 


Ae 2 Ver 2 fae 
0.4342 {1/x?w, +o®/y ee ae /z w, J 
the last step coming from exercise 11 b on page 30. 


_ In this investigation, and in others like it, the S.Es. have 
all been found proportional to the quantities measured; to be specific 


The SE. of x is 7 percent of x 
? vv hi y LS 6 " vr y ; 


ee eee aS. re 8 Mya’ 
It follows, then, from Eq. 13 on ve ae 12 that 
xu, = .(x0/o,)2= 0#/.07” 
ate (yo/oy)*= 07/,06* 
zw 


it 


poe (20/a,)°= ce /.05° 


wherefore 


i/W = (0.434/a)£ {(.07)% + (.06 b)® + (.05 ¢)*} 


which is constant throughout, independent of x, y, and z. This is the 
second of the two important features described above, 


Now o is open to arbitrary choice, since weights are not absolute 
but are relative only (page 12); and a convenient choice is to put 


o2 = 0,4342 {,072 + (.06b)% + (.05¢)*} 
whereupon W becomes unity at all points, The value of o, in this 


problem, is not needed until at the end, when it will be compared with 
o(ext) (sce section 6a, p. 21); what is more important at present, 
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b and c will not be needed for the calculation of W, in spite of the 
fact that x, y, and z are all subject to error. From a computational 
standpoint, this is a fortunate situation, resting on the peculiar 
combination of the form of the fitted function and the S.Es. of x, jy, 
and z. 


As it happens, W being constant (unity) throughout, the same re- 
sults for a, b, and c would come from normal equations set up under 
the (incorrect) assumption that only the measurements on volume are 
subject to error and all of unit weight. But estimates of the parameters, 
however important, are not the whole problem; one ought also to consider 
the adjustment of the observations for a study of the trends (if any) 
in the residuals; one ought also to know the minimized ¢* for considera- 
tions of the fit of the formula, as for example, by comparing o(ext) 
with the prior o, which fortunately is at hand in this example as it 
was also in the preceding one. If the errors in the diameter and mer- 
chantable height are masked, none of the residuals in volume, merchant- 
able hsight, or diameter, can be found; moreover, the entry in Eq. IV 
of the solution, which should be g*, is instead an unknown multiple of 
it, wherefore the possibility of reconciling the known experimental 
conditions with the fit of the curve is lost or put on a basis that is 
apt to do more harm than good, 


Approximate values of a, b, and ec (hence also of a") having been 
found by some method or other (see page 34), or known from previous 
experience, one would next calculate the value of 


a ee! Sight la.” sae A 


at each of the 66 points, the capitals referring to the observed values 
Or Logix ~~. 1, oe y= eee wee el. 


Since W = 1 throughout, tables 1 and 2 of section 16 coalesce, 
and the normal equations are symbolized as follows: 


No, Va" Vb Ve = are | Sum 





I 66 [vy] (2"j - [fo] - 
2 Ly"y" ] [y"z"} zs pte aaa z 
3 LZeZ | e La" ts Z. 
4 Fees - 


Since most of the adjustment is already contained in the approxi- 
mate values of a, b, and c, a.maximum of two figures would suffice in 
any column of X", XY, Z" > or fo} and ya maxinum of three figures would 
likely suffice in the normal equations. Such a simplification is our 
compensation for the trouble of computing f, at each point. 
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The solution would proceed as on page 107, Eq. IV will contain 
the minimized ¢*, correctly distributed among the residuals in volume, 
height, and diameter. The reciprocal matrix found in BOS SP ale eG 
13 will contain the variance and product variance coefficients of a", 
Oy eng c: 


Exercise. Express the variance coefficients of at and a in terms of 
Cc Cc etc. found in the reciprocal matrix for a", b, and c. 


Instead of using approximate values of a, b, and c, and computing 
an f, at each point, Mr. Buell had already adopted the somewhat longer 
process of using a,'= by = Cg = 0, fy = X', and solving for a, b, and 
¢ directly. His normal equations are symbolized as follows, directly 
in terms of the logarithms on page 170, 





No, a baal c = : Sum 
I 66 LYtJ [Za] [x'] = 
ne Ly'yt ] [y'z? J {ty'x' ay Maa 
a Ava yar, - 
4 (x'x'] - 





Numerically, his equations are these: 


No, a b c = L 


i 66 102,451000 84 ,090000 152,136000 
ra 159 .921325 131.022337 nels voage 
3 107 .853544 195.795651 
4 - 056 809522 





The solution was found to be: 


ate 1.76222, a = 0201652 
b = 0,87476 
© 8 2.142206 
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Hence the relation found is 


x = 0.0165 y°o!z,%-14 

Not having at hand the complete form of solution, and in particular, 
not having ¢” as it would appear in the form of solution shown in section 
17 page 107, we shall here make use of exercise 3 of section 17 (see also 
exercise 25 of section 19), thus getting 


iH 


g® = 356,809522 + 152,.136000 - 1.78222 
- 237 .985322 - 0.87476 


—195.790b0L 4) 2.14226 


i 


0.324 


4 


It will be noted that g* is here the small remainder left over 
from the addition and subtraction of relatively much larger numbers. 
To securs two figures in ¢*, one must carry a, b, and ¢ through the 
fourth decimal; this is so in spite of the fact that we can not possibly 
rely statistically on so many figures in a, b, and c, a fact that would 
be evident from their S.Es. or from forest measurements in general. 
This situation is to be contrasted with the relatively few figures that 
would be required for the normal equations if good approximate values 
ao", bo, and co had been used for the calculation of fy at every point; 
with good approximations, the sum [f,f,.] would itself be close to the 
minimized ¢*, so that the correction terms need not be carried far, 
The reader will realize that this matter has been stressed earlier; 
sée e.g> pages 89, 99, 119,120, and 124. 


We can now make the external estimate of o from the value of ¢* 
computed above, using Eq. 21 page 19 with the result that 


o®(ext) = ¢2/(66-3) = 0.324/63 = 0,00514 


This is to be compared with the prior o*, which from the choice made in 
terms of b and c on page 171 turns out to be 


uN 


of = 0,4342 {.072 + (.06 + .875)% + (.05 + 2.142)* } 


It 


0.00356 


Thus o2(ext) is about 50 percent larger than the prior o*, The possibility 
of this comparison is the third feature mentioned at the start. 


A more exact comparison of the two estimates of o can be made as 
follows, First of all, we need ¥*, By Eq. la on page 5, 


y% = g®/o2% = 0.324/0.00356 = 91.6 
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Since tables of chi-square do not run so high as 63 degrees of freedom 
we use Fisher's functicn* 


VO eat ibal® an’ Cle 
which works out to be 2.3, giving P = about 6.02. This is really not 
a bad value of P for an empirical formula of this kind, though it might 


be well to look carefully at the data for inhomogeneities of various 
kinds, - 


It would be interesting to make a study of the residuals as 
functions of x, or y, or z, but we shall not stop here except to indi- 
cate how the residuals would be computed. From Eqs. 87 page 86 we have 


hom Aue Ce te Dey en ZI) at point h 


whence by Eqs. 89 on page 91 the logarithmic residuals are 


Var = (L/wy AN tyr = Afwyr = (0.434-0,07/0)8A 

at 
Vy! = (L/iwy1)r fytist A b/ Wy s = -(0.434°0.06/o)%A, b ie 
Vor = (l/wet)N fz: = - A c/wer = -(0.434°0,05/0)2A ¢ 


since df/dx' = df/dx", etc. 


The special features peculiar -to this problem have been covered, 
and the remaining details will be omitted; the reader, hcwever, will 
profit from Professor Schumacher's comments on the foregoing, which 
commence over the page. 


cm a a me ee ee ee ee ee ee a ee ee 


* This function is written. at the bottom of table III in Fisher's 
Statistical Methods for Research Workers (Oliver and Boyd), all 
editions. When k is large, say above’30, it is distributed very 
nearly as a normal deviate with unit S. D. 
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Comments from Professor Francis X, Schumacher, Duke University 
RR RL = A NSS FP NT NE A TS | LTT ST 


1. The number of figures required in the solution of Mr. Buell'’s 
normal equations could be cut down by the calculation of an f, at every 
point, as you have emphasized, but perhaps a more effectual saving of 
labor would follow upon transferring the origins of coordinates from the 
natural zeros to the logarithmic means X', Y', Z'. We know from the 
first normal equation of either the set at the bottom of page 172 or that 
in the middle of page 173 that the fitted plane will pass through the 
logarithmic means, i, e. the final values of a, b, and ¢ will satisfy 


TON 20M tyres wort 


The transfer of the origins will not only cut down on the number of 
figures required, but will also eliminate the parameter a' and reduce the 
number of normal equations by one, leaving only b and ¢ as the unknowns, 
a' to be found afterward by noting that 


at: Te Reet} ees 
The new sums and cross products (to be denoted by appending ° to the 


brackets) would be found by meking the following reductions from Mr. Buell's 
equations: . 


[y'y'J° = 159,921325 - 102.4512/66 = 0.887880. 
[y’'z']° = 131.022337 - 102.451°84.090/66 = 0.490450 
[Z'z']° = 107,.853544 - 84,090°/66 = 0.715240 
[y'x']° = 237,985322 ~ 102.451°152.136/66 = 1,826453 
[z'x']° = 195,795651 - 84,090°152.136/66 = 1.960557 
[x'x']° = 356,809522 - 152.1362/66 = 6,122215 


Four decimals will suffice, whereupon Mr. Buell's normal equations 
(p. 173) reduce to the following set, which can be solved as shown, 











No, b Cc = 1 sum 
a 0.8879 0.4904 136065 3, 2048 
2 SEES. 1.9666 Oo. L668 
3 Factors GO vLeee 9.9073 
4 -0 55231 -,2708 -1 ,0088 -1.7700 
Ay 4444 0.9518 1.3962/ 
a) -2.05710 ~3,7573 -6.5926 
6 -2.14176 —2,0385 -2.9903 
Lit 0.3264 0.3264/ 
8 b = 0.8742 
7 6} = 2.44138 4) So Alera | 
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The values of b and ¢ just obtained agree well enough with those 


on page 173, but with fewer figures and less trouble; and the same can 
be said for the sum of squares 0.3264 scen in Eq. III. Otherwise obtained, 


6.1222 - 0,.8742°1,8265 - 2.1418°1,9606 


i 


pF 


0.5263 


affording an interesting check. (The two figures 0.3264 and 0.326% for 
~* show a numerical comparison of the two expressions in parts c and a 
respectively of exercise 3 on page 109), 


&. I should like to offer the following in the hope of fostering 
first approximations as a preliminary to the real work of fitting by 
least squares, If the merchantable portion of the tree stem were of the 
same geometrical form in all tree sizes, the volume would vary directly 
with the height and as the square of the diamster. Hence useful approxi- 
mations should be 


b=1 
c= 2 
2) eee ER ee ee a (not needed in the plan 


just outlined) 


The problem is then really one of finding the effect of changes produced 
by the form of the merchantable solid upon tree volume, 
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Adjustment of observations; for- 
mulas, 38, 44, 45, 175; gsome- 
wey, 9O, 96, 168: numerical i1- 
lustration, 165-167, 168; im- 
portance of, 93, 163-166 

Adjustment of parameters 38, 87, 
Dty 4 OO 

Poe RLIKEN TO, 106, 117 

Alternative hypothesis 19, 25 

Analysis of variance 18, 21, - 
24, 25, 164 

Approximate values of parameters 
Gage ote OS; 25S, 160; import- 
Bree.c. 69), 99179, 120, 
hoe tom lL 4, LV, 177 


F. W. BESSEL 2, 42 

Raymond T, BIRGE 18, 20, 116, 
Ly?) 164 

Maxime BOCHER 41 

Jesse H, BUELL 33, 169, 173, 
176 


Calculated curve and calculated 
Pointe, /, 8, 85, 95; see also 
adjustment of observations 

Calculation of mean and S. D. 
Baltr 

Norman CAMPBELL 54 

AwL, CAUCHY 27, 34 

Celestial mechanics, analogy, 
124-125 

Camrose) 125, be8, 130; 
quasi centroid 125 

Chi-square distribution 19, 94 

Cre tes o eee i9, 74, 157, 164, 
169, L7D 

George Cary COMSTOCK 3 

Condition, equations of, Sl, 32, 
52; reduced conditions 355 

Confidence bands for a curve 
Pia-1i6, 167, 165° 

Confidence interval for a func- 
tion 116 

Consistency, see external con- 
sistency 
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Correlates, correlatives 36 

Correlation coefficient 26, 27, 
a Bar: 

Covariance 10 


Harold T. DAVIS 117 

Besse B. DAY 118° 

Degrees of frecdom, history, 18 

W. Edwards DEMING 18, 20, 31, 82, 
SS Gea, Yee LoS 

Abraham De MOIVRE 1, 2 

My Be DeObITrEn Ss 7 <L06 

Arthur Mason DuPRE 1 


J. FE. Encke 2, 42. Lhion 

Equa*ions of condition, see sondi- 

Estimates, unbiased, 18 . 

External consistency 6, 18, 19; 
comparison with internal consis- 
tency, 20, 21, 24, 25; estimate 
of o by external consistency 18, 
LOY G7, LIS VRLOO LOG, eee 
compared with prior o, 19, 68, 
164, 174, 175 


RR. Ay FISHER 15, 2), Vitae ee 
L6e) LO?) P74 Ses wae aod 
t-test 24, 25, L115 

Freezing 105 


W. L. GAINES 143, 150 

Karl Friederich GAUSS 1, 2, SG ,71a, 
66, 42) 75, LODy LOS, gel Gin iaAy 
L25 

Geometrical conditions 52 ff 

Geometrical conditions without 
Lagrange multipliers 46 

General normal equations 37 

Corrado GINI 3, 128 

Meyer A, GIRSHICK 106, 108 

Goodness of fitel9, (el), 157. 164- 
E66 ySE75 


fF. dos S, Haly eg 
J. F. HAYFORD 48 
Harold HOTELLING 106, 108 
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Information (Fisher) 13 

Instability 105, 157, 164; see 
so near indeterminacy 

Indeterminacy, see near indeter- 
minacy 

Internal consistency 20, 159; 
also external consistency 

Inverse matrix, see. reciprocal 
matrix ¥ 

Isotherm 153 


al- 


see 


Truman L. KELLEY 106 

C. KRAMP 1 

Charles H, KUMMELL 2, 42, 47 

J. L. LAGRANGE 36; see also La- 
grange multipliers 

Lagrange multipliers 2, 35, 3, 
41, 67, 78; solution without 
Lagrange multipliers 46, 47 

Least squares, principle of, 5, 6, 
Oe, £5, 93s definition of the 
method of, 6; fundamental no- 
CLOUse ae ol, oewnow, Shee 

O. M. LELAND 48, 103, 105 

“Line of.worst fit 128 

Logarithmic form, special remarks, 
143-145 

Paul LORENZ 118 


Matrix, reciprocal; 
matrix 

Mean 8, 9, 31, 34 

Mean square errar, propagation, ae 

Mansfield MERRIMAN 2 

Method of averages 34 

Method of least squares, 
tion, 6 

Method of selected points 34, 160 

A. MICHELS 153 


see reciprocal 


defini- 


Near indeterminacy 105, 111, 121,135 
Viva, NERRASS OUR 11S 

J. NEYMAN 18, 116, 167 

Max NORDAU 1 


Normal equations, systematic solu- 
tion--simple illustrations 10, 
24, 49, 90; more complicated il- 
lustrations 64, 65, 155,° tee, 
exhibit in symbols 107; forma- 
tion 52 ff, 99 ff5 positive liam 
inite 41, 87; general 37; geo- 
metrical conditions 41; curve 
fitting 86, 87; freezing 105 

K, A. NORTON 143 

Number of figures cut down by good 
approximetions 89, .99, 119, )ieGs 
124, 0127 se, See 


A. de Forest PALMER 18, 105, 164 
Parameters not independent 153; 
Significance of, 24, 116; weights 
and variance coefficients of, le, 
26, 113, lel, 1é2e, 155). [OG geeee 
Parabola, special methods, 117. 
Karl PEARSON 1, 2, 18, 128, 1350 
Powers of 10, manipulation, 62, 63, 
64. £f;. 102) 156, 162 petee 
Principle of. least squares 5, 6, 
32, 45, 93 
Prior value of o 159, 163, 169, 
171, 174; compared with o(ext), 
19; 66, (164 Slee, oles 
Product variance 10 
Propagation of error 27-30, 155, 
167; of mean square error 27; of 
variance 27 
Ruth R. PUFFER, see H, B. Wilson .- 
Quasi centroid 125 


Reciprocal matrix 10, 75, 76, 104, 
105, 106, 3106) 109°. Lie a tee 
129, 295 7.205.. 158.) Gent cee 

Reciprocal solution 105, lll, ida; 
156, 157, 164; see also near 
indeterminacy ay 

Reducsd conditions 55 

Lowell J. REED 3 


Peeinuals 5, 7, 16, 31,34, 36, 
83; calculation 36, 91, 165- 
Paes, Sun ot, 1S) 17... 120 
126, 127, 129 [Robinson 

G, ROBINSON, see Whittaker and 


? 


Frank S, SALISBURY 106 

Max SASULY 117 

Henry SCHULTZ 114-116 

Francis X, SCHUMACHER 169, 
176-177 

J. B. SCARBOROUGH 18 

Second adjustment 33, 124, 125 

selected points, method of, 4, 
160 

John D, SHEA 117 

Walter A, SHEWHART 22, 116 

T. SMITH 106 

Solution unequivocal] 103, 143 

Standard deviation (S.D.) of a 
sample 8 

Solution without Lagrange mul- 
tipliers 46, 47 ; 

Standardized residuals 13, 14 

Standard errors of parameters 
ana functions 20, 24, lel, 
Boe, LOO 

R. Meldrum STEWART 3, 42 

Frederick F, ST#PHAN 117 

STUDENT 115, 116 

Bumecheck 54, 63, 64, 65, 101, 
DO doo loa, L7G 

Sum of squares (#7) 109-111, 
eevee, let, Leo, 132, 134, 
toe, Loew .o/, 165; 
short expression, 40, 109-111; 
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removed by regression, 120; 
special formulas for a line, 
LOU -tol 

Systematic solution, see normal 
equations 


t-test and t integral 24, 116 

Tests of significance 19, 21, 24, 
OO ele: 

True curve and true points Figs. 8 
Gio, Do, Gere 

L. B. TUCKERMAN 157, 164 


Horace S. UHLER 3, 42 
Unbiased estimates 18, 113, 122 


Variance coefficient (= 1/weight) 
Ll, Leyva, 115 
Variance, propagation of, 27 


Weights, definition, 12,.15, 30; 
weights of functions ll, 20, 
24, 47 £f, 60 If, 69, 69, 74; 
short method in geometrical 
conditions 75 ff; weights of 
parameters 12,°26,° 113 etc, 
Les, LOS fe Loo eo 

H. T. WENSEL 164 

E. TT. WHITPAKER and G. ROBIN-= 
SON 1047-106, 10S sire 

Benjamin WILLIAMSON 356 

E. B. WILSON 12 

T. W. WRIGHT 48 


z (Fisher) 19, 21; see also anal- 
ysis of variance 

















